Arize AI
Search…
⌃K

Custom Metric Examples

Common example use cases

Example Custom Metrics

Custom metrics are an extremely powerful tool to evaluate many dimensions of your ML model. From analyzing the business impact of your model to calculating a moving average, you can leverage custom metrics in many ways. Use this page as a guide on how you can create custom metrics tailored to your ML needs.

Percent of Features With Value > X

In this example, we have a lending model. Assume that the lending model automatically rejects customers with fico scores < 600. You want to track the percent of customers applying for loans with a fico score less than 600, in order to track the health of inbound applications. We can use a FILTER (WHERE ... ) clause to only filter on the first COUNT(*) and not the second.
SELECT
COUNT(*) FILTER (WHERE fico_score < 600) / COUNT(*) * 100
FROM model
Learn more about FILTER (WHERE) clauses here.

Aggregated Business Metrics

Models often take business inputs as features and output decisions. You may want to monitor or dashboard business KPIs that are a combination of these inputs and outputs.
In this example, we have an inventory prediction model. numericPredictionLabel is the predicted sales. We want to calculate excess inventory, defined as the predicted sales minus actual sales minus the current inventory, which is defined by current_inventory.
SELECT
AVG(numericPredictionLabel - numericActualLabel - current_inventory)
FROM model
Learn more about AVG and other aggregation functions here.

Weighted Average

Weighted averages can be useful when different predictions have different importance to the business.
Here we have a model that predicts fraudulent transactions. We want to know what percent of loans in dollar terms are expected to be fraudulent. We can do this with a FILTER clause.
SELECT
(
COUNT(*) FILTER (WHERE categoricalPredictionLabel = 'fraud')
) * feature.loan_amount / SUM(loan_amount) * 100
FROM model

Count Distinct With Filter

In this example, we have a fraudulent login detection model. This model makes one prediction per login, meaning a single user could be part of multiple predictions. We want to count what percent of users have had one fraudulent login.
SELECT
APPROX_COUNT_DISTINCT("user_id") FILTER (WHERE categoricalPredictionLabel = 'fraud') /
APPROX_COUNT_DISTINCT("user_id")
FROM model
Learn more about APPROX_COUNT_DISTINCT and other aggregation functions here.

Calculate Precision, and F beta

The performance metrics in the Arize platform have been replicated as functions in custom metrics. These functions give you greater flexibility in choosing different prediction, actual, and positive class values.

Precision

PRECISION ()
When a model has multiple predictions that are sent as features or tags, the PRECISION function allows you to specify the exact columns used. In this example, a user may have a model with multiple predictions, each one sent in as a tag or feature. The PRECISION() function allows you to specify different columns to use for predictions or actuals, using the predicted and actual keyword arguments:
SELECT
PRECISION(
predicted=multi_output_prediction,
actual=multi_output_actual,
pos_class='custom_positive_class'
)
FROM model
Learn more about PRECISION and related functions here.

F Beta

F1 score is a special case of F beta, where precision and recall are equally weighted. In some use cases, you may prefer to weight precision more than recall, or vice versa, based on business needs. The F_BETA function allows you to specify the beta parameter to adjust this weighting. In this example, we want to weigh precision twice as much as recall, with a beta score of 0.5.
SELECT
F_BETA(beta=0.5)
FROM model
Learn more about F_BETA and related functions here.

Pinball Loss Function

Custom metrics can be used to create performance metrics such as pinball loss. Pinball loss is useful for quantile forecasts. An example could be a weather prediction, such as an 80% chance of 20 degree or colder weather.
Let y be the actuals, and z be the prediction. α is the quantile, in this case, 0.8 (for 80% chance). The formula for pinball loss is as such.
L(y, z) = (y - z) * α if y >= z
= (z - y) * (1 - α) if z > y
This is what pinball loss would look like for a numeric model.
SELECT
AVG(
CASE
WHEN numericActualLabel >= numericPredictionLabel
THEN (numericActualLabel - numericPredictionLabel) * 0.8
ELSE (numericPredictionLabel - numericActualLabel) * (1 - 0.8)
END
)
FROM model
Custom metrics are not just limited to predictions and actuals. For example, if multiple predictions at multiple quantiles are sent in as tags, such as a tag prediction_temp_p80 for the 80th percentile temperature prediction, pinball loss could look like this:
SELECT
AVG(
CASE
WHEN numericActualLabel >= prediction_temp_p80
THEN (numericActualLabel - prediction_temp_p80) * 0.8
ELSE (numericActualLabel - prediction_temp_p80) * (1 - 0.8)
END
)
FROM model
Learn more about CASE statements and other conditional structures here.