Multi-Class Classification

How to log your model schema for multiclass classification models

Multi-Class Classification Overview

A classification model with more than two classes.

Supported Metrics

Micro-Averaged Precision, Micro-Averaged Recall, Macro-Averaged Precision, Macro-Averaged Recall, Precision for a Class, Recall for a Class

How To Log Multi-Class Data

Log multi-class classification models based on your use case

Use Case

Description

Expected Fields

Single-Label

A prediction that has 1 label i.e. A passenger can only be in EITHER economy, business, OR first-class

prediction scores (dictionary)
actual scores (dictionary, optional)

Multi-Label

A prediction that has multiple labels i.e. A song can be multiple genres such as 'pop-rock'

prediction scores (dictionary)
threshold scores (dictionary)
actual scores (dictionary, optional)

Single-Label Use Case

Example Row

prediction_scores

actual_scores

[{"class_name": "economy_class", "score": 0.81},{"class_name": "business_class", "score": 0.42},{"class_name": "first_class", "score": 0.35}]

[{"class_name": "economy_class", "score": 1}]

Note: class economy_class has the highest prediction score and will be the prediction label

Code Example

schema = Schema(
    prediction_id_column_name="prediction_id",
    prediction_score_column_name="prediction_scores",
    actual_score_column_name="actual_scores",
)

response = arize_client.log(
    model_id='multiclass-classification-single-label-example', 
    model_version= "v1",
    model_type=ModelTypes.MULTI_CLASS,
    dataframe=example_dataframe,
    schema=schema,
    environment=Environments.PRODUCTION,
)

# Predicting likelihood of Economy, Business, or First Class
"""
example_record = {
   "prediction_scores":{
      "economy_class":0.81,
      "business_class":0.42,
      "first_class":0.35
   },
   "actual_scores": {
      "first_class": 1
   }
}
"""
prediction_label = MultiClassPredictionLabel(
	prediction_scores=record["prediction_scores"],
)
actual_label = MultiClassActualLabel(
	actual_scores=record["actual_scores"],
)
response = arize_client.log(
    model_id="multiclass-classification-single-label-example",
    model_version= "v1",
    model_type=ModelTypes.MULTI_CLASS,
    prediction_id=record["prediction_id"],
    prediction_label=prediction_label,
    actual_label=actual_label,
    environment=Environments.PRODUCTION
)

Note: class economy_class has the highest prediction score and will be the prediction label

Multi-Class Prediction

Log multi-class prediction values using the MultiClassPredictionLabel object, which can be assigned to the prediction_label parameter.

Prediction scores are required to use MULTI_CLASS models in Arize.

class MultiClassPredictionLabel(
    prediction_scores: Dict[str, Union[float, int]]
)

Multi-Class Actual Values

Log multi-class actual values by using the MultiClassActualLabel object, which can be assigned to the actual_label parameter.

Actual values are optional, and can be sent later via delayed actuals.

class MultiClassActualLabel(
    actual_scores: Dict[str, Union[float, int]]
)

For more information on Python Single Record Logging API Reference, visit here:

Single Record Logging

Multi-Label Use Case

Example Row

prediction_scores

threshold_scores

actual_scores

[{"class_name": "jazz", "score": 0.81},{"class_name": "rock", "score": 0.42},{"class_name": "pop", "score": 0.35}]

[{"class_name": "jazz", "score": 0.5},{"class_name": "rock", "score": 0.4},{"class_name": "pop", "score": 0.6}]

[{"class_name": "rock", "score": 1}]

Note: classes jazz and rock have prediction scores > threshold scores and will be part of the prediction label.

Code Example

schema = Schema(
    prediction_id_column_name="prediction_id",
    prediction_score_column_name="prediction_scores",
    multi_class_threshold_scores_column_name="threshold_scores"
    actual_score_column_name="actual_scores",
)

response = arize_client.log(
    model_id='multiclass-classification-multi-label-example', 
    model_version= "v1",
    model_type=ModelTypes.MULTI_CLASS,
    dataframe=example_dataframe,
    schema=schema,
    environment=Environments.PRODUCTION,
)

#predicting song genre within pop, rock, and jazz categories
"""
example_record = {
"prediction_id": "57437c5d-0cd2-48bf-aa91-379302fd0fe4",
"prediction_scores": {
   "pop": 0.5044088627000001,
   "rock": 0.3688503145,
   "jazz": 0.1267408228
 },
 "threshold_scores": {
   "pop": 0.50,
   "rock": 0.40,
   "jazz": 0.10
 },
"actual_scores": {
   "pop": 1,
   "jazz": 1
 }
"""

prediction_label = MultiClassPredictionLabel(
	prediction_scores=record["prediction_scores"],
        threshold_scores=record["threshold_scores"],  #required for multi-label
)
actual_label = MultiClassActualLabel(
	actual_scores=record["actual_class"],
)
response = arize_client.log(
    model_id="multiclass-classification-multi-label-example",
    model_version= "v1",
    model_type=ModelTypes.MULTI_CLASS,
    prediction_id=record["prediction_id"],
    prediction_label=prediction_label,
    actual_label=actual_label,
    environment=Environments.PRODUCTION
)

Multi-Class Prediction & Threshold Values

Log multi-class prediction and threshold values using the MultiClassPredictionLabel object, which can be assigned to the prediction_label parameter.

Prediction scores are required to use MULTI_CLASS models in Arize.

class MultiClassPredictionLabel(
    prediction_scores: Dict[str, Union[float, int]]
    threshold_scores: Dict[str, Union[float, int]] = None #for multi-label use cases
)

Multi-Class Actual Values

Log multi-class actual values by using the MultiClassActualLabel object, which can be assigned to the actual_label parameter.

Actual values are optional, and can be sent later via delayed actuals.

class MultiClassActualLabel(
    actual_scores: Dict[str, Union[float, int]]
)

For more information on Python Single Record Logging API Reference, visit here:

Single Record Logging

Inferring Labels From Uploaded Scores

To calculate metrics and visualize & troubleshoot data for multi-class models, Arize automatically infers prediction & actual labels from the scores that you upload.

Learn how each case is determined below.

Use Case

Prediction Label Determination

Actual Label Determination

Single-Label

For each prediction, the class with the highest prediction score is the prediction label

The class with an actual score of 1 is the actual label

Multi-Label

For each class, there must exist a prediction score and threshold score. If the prediction score > threshold score, the class is a part of the prediction label

Each class with an actual score of 1 is part of the actual label

PreviousInsurance NextRegression

Last updated 8 months ago