Binary Classification

How to log your model schema for binary classification models

Binary Classification Cases

Binary Classification Cases	Expected Fields	Performance Metrics
Case 1: Supports Only Classification Metrics	prediction label, actual label	Accuracy, Recall, Precision, FPR, FNR, F1, Sensitivity, Specificity
Case 2: Supports Classification, AUC, Log Loss Metrics	prediction score, prediction label, actual label	AUC, PR-AUC, Log Loss, Accuracy, Recall, Precision, FPR, FNR, F1, Sensitivity, Specificity
Case 3: Supports AUC & Log Loss Metrics	prediction score, actual label	AUC, PR-AUC, Log Loss

Click here for all valid model types and metric combinations.

Case #1 - Supports Only Classification Metrics

Python Pandas Batch

Google Colaboratory

Example Row

state	pos_approved	zip_code	age	prediction_label	actual_label	prediction_ts
ca	True	12345	25	not_fraud	fraud	1618590882

Code Example

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_label_column_name="prediction_label",
    actual_label_column_name="actual_label",
    feature_column_names=["state", "pos_approved"],
    tag_column_names=tag_columns,
)

response = arize_client.log(
    model_id='sample-model-1', 
    model_version='v1', 
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    metrics_validation=[Metrics.CLASSIFICATION],
    environment=Environments.PRODUCTION,
    dataframe=example_dataframe,
    schema=schema
)

For more details on Python Batch API Reference, visit here:

Pandas Batch Logging

Python Single Record

Google Colaboratory

Code Example

# features & tags can be optionally defined with typing
features = {
    'state': 'ca',
    'pos_approved': TypedValue(value=True, type=ArizeTypes.INT)
}
    
tags = {
    'zip_code': '12345',
    'age': '25'
}

response = arize_client.log(
    model_id='sample-model-1', 
    model_version='v1', 
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    environment=Environments.PRODUCTION,
    features=features,
    tags=tags,
    prediction_label="not fraud",
    actual_label="fraud"
)

For more information on Python Single Record Logging API Reference, visit here:

Single Record Logging

Data Connector

Download an example Parquet file: Open parquet reader here.

https://storage.cloud.google.com/arize-assets/documentation-sample-data/data-ingestion/binary-classification-assets/binary_classification_metrics_only_batch_ingestion_data.parquetstorage.cloud.google.com

Learn how to upload files via various Data Connectors:

Google Cloud Storage (GCS)AWS S3Azure Blob StorageGoogle BigQuery

Case #2 - Supports Classification & AUC/Log Loss Metrics

Python Pandas

Google Colaboratory

Example Row

state	pos_approved	zip_code	age	prediction_label	actual_label	prediction_score	prediction_tsa
ca	True	12345	25	not_fraud	fraud	0.3	1618590882

Code Example

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_label_column_name="prediction_label",
    prediction_score_column_name="prediction_score",
    actual_label_column_name="actual_label",
    feature_column_names=["state", "pos_approved"],
    tag_column_names=tag_columns,
)

response = arize_client.log(
    model_id='sample-model-1', 
    model_version='v1', 
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    metrics_validation=[Metrics.CLASSIFICATION, Metrics.AUC_LOG_LOSS],
    environment=Environments.PRODUCTION,
    dataframe=test_dataframe,
    schema=schema,
)

For more details on Python Batch API Reference, visit here:

Pandas Batch Logging

Python Single Record

Google Colaboratory

Code Example

features = {
    'state': 'ca',
    'pos_approved': True,
    'item_count': 10
}
# features & tags can be optionally defined with typing 
tags = {
    'zip_code': TypedValue(value='12345', type=ArizeTypes.INT),
    'age': '25'
}

response = arize_client.log(
    model_id='sample-model-1', 
    model_version='v1', 
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    environment=Environments.PRODUCTION,
    features=features,
    tags=tags,
    prediction_label=("not fraud", 0.3),
    actual_label="fraud"
)

For more information on Python Single Record Logging API Reference, visit here:

Single Record Logging

Data Connector

Download an example Parquet file: Open parquet reader here.

https://storage.cloud.google.com/arize-assets/documentation-sample-data/data-ingestion/binary-classification-assets/binary_classification_auc_log_loss_batch_ingestion_data.parquetstorage.cloud.google.com

Learn how to upload files via various Data Connectors:

Google Cloud Storage (GCS)AWS S3Azure Blob StorageGoogle BigQuery

Case #3: Supports AUC & Log Loss Metrics

Python Pandas

Example Row

state	pos_approved	zip_code	age	actual_label	prediction_score	prediction_ts
ca	True	12345	25	fraud	0.3	1618590882

Code Example

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_score_column_name="prediction_score",
    actual_label_column_name="actual_label",
    feature_column_names=["state", "pos_approved"],
    tag_column_names=tag_columns,
)

response = arize_client.log(
    model_id='sample-model-1', 
    model_version='v1', 
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    metrics_validation=[Metrics.CLASSIFICATION, Metrics.REGRESSION, Metrics.AUC_LOG_LOSS],
    environment=Environments.PRODUCTION,
    dataframe=test_dataframe,
    schema=schema,
)

For more details on Python Pandas API Reference, visit here:

Pandas Batch Logging

Python Single Record

To declareprediction_score ONLY, pass a tuple of an empty string and your prediction_score through the prediction_label argument.

Code Example

# features & tags can be optionally defined with typing
features = {
    'state': 'ca',
    'pos_approved': TypedValue(value=True, type=ArizeTypes.INT)
}  

tags = {
    'zip_code': '12345',
    'age': '25'
}
    
response = arize_client.log(
    model_id='sample-model-1', 
    model_version='v1', 
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    environment=Environments.PRODUCTION,
    features=features,
    tags=tags,
    prediction_label=("", 0.3),
    actual_label="fraud",
)

For more information on Python Single Record Logging API Reference, visit here:

Single Record Logging

Default Actuals

For some use cases, it may be important to treat a prediction for which no corresponding actual label has been logged yet as having a default negative class actual label.

For example, consider tracking advertisement conversion rates for an ad clickthrough rate model, where the positive class is click and the negative class is no_click. For ad conversion purposes, a prediction without a corresponding actual label for an ad placement is equivalent to logging an explicit no_click actual label for the prediction. In both cases, the result is the same: a user has not converted by clicking on the ad. For AUC-ROC, PR-AUC, and Log Loss performance metrics, Arize supports treating predictions without an explicit actual label as having the negative class actual label by default. In the above example, a click prediction without an actual would be treated as a false positive, because the missing actual for the prediction would, by default, be assigned to the no_click negative class.

This feature can be enabled for monitors and dashboards via the model performance config section of your model's config page.

Quick Definitions

Prediction Label: The classification label of this event (Cardinality = 2)

Actual Label: The ground truth label (Cardinality = 2)

Prediction Score: The likelihood of the event (Probability between 0 to 1)

Actual Score: The ground truth score (0 or 1)