Arize AI
Search…
3. Model Type
Arize supports performance metrics for various Model Types.

Overview

Arize supports 2 model types:
  • Numeric/Regression Models - Models that have continuous, numeric outputs
  • Score Categorical Models - Models that have a discrete classification label (optionally also a score value)
Example: Regression prediction value = 82°F; Score Categorical prediction score = 0.82, prediction label = hot
Depending on the model type, your prediction and actual fields format will be varied.
Classification
Numeric/Regression
Ranking

Classification Models

Classification models predict categories. Arize supports logging the prediction category and the prediction score.

Code Example

1
response = arize.log(
2
model_id='sample-model-1',
3
model_version='v1",
4
model_type=ModelTypes.SCORE_CATEGORICAL,
5
...
6
prediction_label="not fraud",
7
prediction_score=0.3,
8
actual_label="fraud",
9
actual_score=0.6
10
)
Copied!
Field
Data Type
Prediction Label
string
Prediction Score (optional)
float
Actual Label
string
Actual Score (optional)
float

Example ML Use Cases

  • Fraud Models
    • Prediction Label: Is this a Fraud Transaction? Label = "fraud"
    • Prediction Score: How likely is this to be fraud? -> Score = 0.4
    • Actual Label: "Was this actually a fraud transaction?" Label = "not fraud"
    • Actual Score: "Was this actually fraud?" Score = 0

Performance Metrics Supported

If you include score values, this model type will support both classification and regression metrics.
For some use cases, it may be important to treat a prediction for which no corresponding actual label has been logged yet as having a default negative class actual label. For example, consider tracking advertisement conversion rates for an ad clickthrough rate model, where the positive class is click and the negative class is no_click. For ad conversion purposes, a prediction without a corresponding actual label for an ad placement is equivalent to logging an explicit no_click actual label for the prediction. In both cases, the result is the same: a user has not converted by clicking on the ad. For AUC-ROC, PR-AUC, and Log Loss performance metrics, Arize supports treating predictions without an explicit actual label as having the negative class actual label by default. In the above example, a click prediction without an actual would be treated as a false positive, because the missing actual for the prediction would by default be assigned to the no_click negative class.
This feature can be enabled for monitors and dashboards via the model performance config section of your model's config page.

Numeric/Regression Models

Regression (Numeric) models are used when the objective is to predict a specific number value based on a set of features -- e.g. ETA of a food delivery, or predicted house price based on home age, square footage, etc. Performance metrics are calculated by using the predicted value and the actual value.

Code Example

1
response = arize.log(
2
model_id='sample-model-1',
3
model_version='v1',
4
model_type=ModelTypes.NUMERIC,
5
...
6
prediction_value=100,
7
actual_value=90
8
)
Copied!

Field
Data Type
Prediction Label
Numeric (Float/Int)
Actual Label
Numeric (Float/Int)

Example ML Use Cases

  • Demand Forecasting
    • Prediction Label: "What is the estimated demand in this region?" Ex: 1000
    • Actual Label: "What was the actual demand in this region?" Ex: 500

Performance Metrics Supported

  • MAPE: Mean Absolute Percent Error
  • MAE: Mean Absolute Error
  • RMSE: Root Mean Square Error
  • MSE: Mean Squared Error
  • R^2: R-Squared or Coefficient of Determination
  • Mean Error: Average Error between Predictions & Actuals

Ranking Models

Ranking models are used by search engines or other information retrieval systems to display query results ranked in the order of the highest relevance. To assess the performance of ranking models, Arize supports logging the sequence of actual relevance scores in the order of the displayed results.

Code Example

This example uses the pandas logger. relevance_scores is the name of a pandas DataFrame column where each row is a list of numbers (or None if a list is not applicable). See this Colab for details.
1
schema = Schema(
2
prediction_id_column_name="prediction_id",
3
...
4
prediction_label_column_name="prediction_label",
5
actual_label_column_name="actual_label",
6
actual_numeric_sequence_column_name="actual_relevance_scores",
7
)
8
response = arize_client.log(
9
model_id="sample-model-1",
10
model_version= "v1",
11
model_type=ModelTypes.SCORE_CATEGORICAL,
12
environment=Environments.PRODUCTION,
13
dataframe=test_dataframe,
14
schema=schema,
15
)
Copied!
Field
Data Type
Prediction Label
string
Prediction Score (optional)
float
Actual Label
string
Actual Score (optional)
float
Actual Numeric Sequence (optional)
List of floats on the first query result; None on additional query results (if any)

Example ML Use Cases

  • Ranking Models
    • Prediction Label: Set to "relevant" (since only relevant results are displayed).
    • Actual Label: Set to "relevant" or "not relevant" for the first search result, and similarly for each search result if also logging additional results of the same query.
    • Actual Numeric Sequence: Set to a sequence of actual relevance scores, e.g [0, 1, 0, 1], on the first query result. If additional results of the same query are also logged, it should be set to None.

Performance Metrics Supported

If you include the sequence of actual relevance scores, the following metric is supported.
Questions? Email us at [email protected] or Slack us in the #arize-support channel
Copy link