# Custom Metric Examples

Common example use cases

## Example Custom Metrics

Custom metrics are an extremely powerful tool to evaluate many dimensions of your ML model. From analyzing the business impact of your model to calculating a moving average, you can leverage custom metrics in many ways. Use this page as a guide on how you can create custom metrics tailored to your ML needs.

The following examples use the `arize-demo-fraud-use-case` model to calculate business KPIs and other useful statistics using custom metrics.

### Percent of Features With Value > X

We'll use this example to calculate the percentage of predictions with a fico score below 600. Assume that the lending model automatically rejects customers with scores below 600, so you want to track this metric to determine the health of inbound applications. We can do this by using a `FILTER (WHERE ... )` clause, which allows us to apply the < 600 filter to only the numerator and not the denominator.
SELECT
COUNT(*) FILTER (WHERE fico_score < 600) / COUNT(*) * 100
FROM model
Learn more about `FILTER (WHERE)` clauses here.

### Weighted Average Performance Metric

Weighted averages can be useful when different predictions are important to your business.
To do this, we'll calculate recall but weigh the predictions by the loan amount. This way, a false negative on more expensive loans will have a higher impact on the score than a false negative on a cheaper loan.
We can use a `WHERE` clause to filter for the actual fraud cases, and a `FILTER (WHERE ... )` clause to filter the numerator where the model incorrectly predicted `not_fraud` .
SELECT
SUM(loan_amount) FILTER (WHERE categoricalPredictionLabel = 'fraud') /
SUM(loan_amount)
FROM model
WHERE categoricalActualLabel = 'fraud'

### Count Distinct

You can use `APPROX_COUNT_DISTINCT` to get the cardinality of a dimension. In this example, we can calculate the average total loan amount per merchant.
SELECT
SUM(loan_amount) / APPROX_COUNT_DISTINCT(merchant_ID)
FROM model
Learn more about `APPROX_COUNT_DISTINCT` and other aggregation functions here.

## Performance metrics

Use natively supported performance metrics as a function that can take multiple arguments for enhanced performance monitoring flexibility. Additionally, create completely new metrics using conditionals and other logic shown below.
View the documentation for our performance metrics here.

### Precision

`PRECISION ()`
When a model has multiple predictions that are sent as features or tags, the `PRECISION` function allows you to specify the exact columns used. In this example, a user may have a model with multiple predictions, each one sent in as a tag or feature. The PRECISION() function allows you to specify different columns to use for predictions or actuals, using the `predicted` and `actual` keyword arguments:
SELECT
PRECISION(
predicted=multi_output_prediction,
actual=multi_output_actual,
pos_class='custom_positive_class'
)
FROM model
Learn more about `PRECISION` and related functions here.

### Percent Error of Model > X

Calculate the percent error to compare your model's predictions over actual values. In this example, we'll create a custom metric to calculate the percent error greater than 9%.
select avg(
case
when abs(scorePredictionLabel - scoreActualLabel)/scoreActualLabel > 9
then 1
else 0
end)
from model
where scorePredictionLabel is not null
and scoreActualLabel is not null
and scoreActualLabel != 0

### Choosing a different threshold for classification metrics

By combining classification metrics with case statements, users can use a different classification threshold than the currently ingested data.
To do this, you can use the `predicted` argument of the metric function. This expects the prediction label, which you can conditionally set using a `CASE` statement.
This works for all classification metrics with the `predicted` argument.
SELECT
PRECISION(
predicted=(CASE WHEN scorePredictionLabel > 0.25 THEN 'fraud' ELSE 'not_fraud' END),
pos_class='fraud'
)
FROM model

### F Beta

F1 score is a special case of F beta, where precision and recall are equally weighted. In some use cases, you may prefer to weigh precision more than recall, or vice versa, based on business needs. The `F_BETA` function allows you to specify the `beta` parameter to adjust this weighting. In this example, we want to weigh precision twice as much as recall, with a beta score of 0.5.
SELECT
F_BETA(beta=0.5)
FROM model
Learn more about `F_BETA` and related functions here.

### Pinball Loss Function

Custom metrics can be used to create performance metrics such as pinball loss. Pinball loss is useful for quantile forecasts. An example could be a weather prediction, such as an 80% chance of 20 degrees or colder weather.
Let y be the actuals and z be the prediction. α is the quantile, in this case, 0.8 (for 80% chance). The formula for pinball loss is as such.
L(y, z) = (y - z) * α if y >= z
= (z - y) * (1 - α) if z > y
This is what pinball loss would look like for a numeric model.
SELECT
AVG(
CASE
WHEN numericActualLabel >= numericPredictionLabel
THEN (numericActualLabel - numericPredictionLabel) * 0.8
ELSE (numericPredictionLabel - numericActualLabel) * (1 - 0.8)
END
)
FROM model
Custom metrics are not just limited to predictions and actuals. For example, if multiple predictions at multiple quantiles are sent in as tags, such as a tag `prediction_temp_p80` for the 80th percentile temperature prediction, pinball loss could look like this:
SELECT
AVG(
CASE
WHEN numericActualLabel >= prediction_temp_p80
THEN (numericActualLabel - prediction_temp_p80) * 0.8
ELSE (numericActualLabel - prediction_temp_p80) * (1 - 0.8)
END
)
FROM model
Learn more about `CASE` statements and other conditional structures here.

## Data Consistency

Data consistency measures the discrepancies between two datasets (i.e. online versus offline features). To compare the same set of features, we recommend sending in one dataset as `feature` values and the other dataset as `tag` values.
For support with creating data consistency metrics, reach out to [email protected] or message us on Slack here.

### Match Rate

Percent of unmatched feature values for a given column.
SELECT COUNT(*) FILTER(WHERE (online_feature != offline_feature)) / COUNT(*) * 100
FROM model

### Average Residual Error

The average difference between two datasets for a given column.
SELECT AVG(ABS(offline_feature - online_feature))
FROM model