Binary Classification

How to log your model schema for binary classification models

Binary Classification Cases

Binary Classification CasesExpected FieldsPerformance Metrics

prediction label, actual label

Accuracy, Recall, Precision, FPR, FNR, F1, Sensitivity, Specificity

prediction score, prediction label, actual label

AUC, PR-AUC, Log Loss, Accuracy, Recall, Precision, FPR, FNR, F1, Sensitivity, Specificity

prediction score, actual label

AUC, PR-AUC, Log Loss

Click here for all valid model types and metric combinations.

Case #1 - Supports Only Classification Metrics

Example Row

statepos_approvedzip_codeageprediction_labelactual_labelprediction_ts

ca

True

12345

25

not_fraud

fraud

1618590882

Code Example

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_label_column_name="prediction_label",
    actual_label_column_name="actual_label",
    feature_column_names=["state", "pos_approved"],
    tag_column_names=tag_columns,
)

response = arize_client.log(
    model_id='sample-model-1', 
    model_version='v1', 
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    metrics_validation=[Metrics.CLASSIFICATION],
    environment=Environments.PRODUCTION,
    dataframe=example_dataframe,
    schema=schema
)

For more details on Python Batch API Reference, visit here:

Pandas Batch Logging

Case #2 - Supports Classification & AUC/Log Loss Metrics

Example Row

statepos_approvedzip_codeageprediction_labelactual_labelprediction_scoreprediction_tsa

ca

True

12345

25

not_fraud

fraud

0.3

1618590882

Code Example

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_label_column_name="prediction_label",
    prediction_score_column_name="prediction_score",
    actual_label_column_name="actual_label",
    feature_column_names=["state", "pos_approved"],
    tag_column_names=tag_columns,
)

response = arize_client.log(
    model_id='sample-model-1', 
    model_version='v1', 
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    metrics_validation=[Metrics.CLASSIFICATION, Metrics.AUC_LOG_LOSS],
    environment=Environments.PRODUCTION,
    dataframe=test_dataframe,
    schema=schema,
)

For more details on Python Batch API Reference, visit here:

Pandas Batch Logging

Case #3: Supports AUC & Log Loss Metrics

Example Row

statepos_approvedzip_codeageactual_labelprediction_scoreprediction_ts

ca

True

12345

25

fraud

0.3

1618590882

Code Example

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_score_column_name="prediction_score",
    actual_label_column_name="actual_label",
    feature_column_names=["state", "pos_approved"],
    tag_column_names=tag_columns,
)

response = arize_client.log(
    model_id='sample-model-1', 
    model_version='v1', 
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    metrics_validation=[Metrics.CLASSIFICATION, Metrics.REGRESSION, Metrics.AUC_LOG_LOSS],
    environment=Environments.PRODUCTION,
    dataframe=test_dataframe,
    schema=schema,
)

For more details on Python Pandas API Reference, visit here:

Pandas Batch Logging

Default Actuals

For some use cases, it may be important to treat a prediction for which no corresponding actual label has been logged yet as having a default negative class actual label.

For example, consider tracking advertisement conversion rates for an ad clickthrough rate model, where the positive class is click and the negative class is no_click. For ad conversion purposes, a prediction without a corresponding actual label for an ad placement is equivalent to logging an explicit no_click actual label for the prediction. In both cases, the result is the same: a user has not converted by clicking on the ad. For AUC-ROC, PR-AUC, and Log Loss performance metrics, Arize supports treating predictions without an explicit actual label as having the negative class actual label by default. In the above example, a click prediction without an actual would be treated as a false positive, because the missing actual for the prediction would, by default, be assigned to the no_click negative class.

This feature can be enabled for monitors and dashboards via the model performance config section of your model's config page.

Quick Definitions

Prediction Label: The classification label of this event (Cardinality = 2)

Actual Label: The ground truth label (Cardinality = 2)

Prediction Score: The likelihood of the event (Probability between 0 to 1)

Actual Score: The ground truth score (0 or 1)

Last updated

Copyright © 2023 Arize AI, Inc