Custom Metrics Query Language

Define metrics tailored to your ML needs

Custom Metrics Overview

Custom metrics allow you to define a metric tailored to your ML needs. Define metrics such as business KPIs, custom performance calculations, weighted averages, etc. Use custom metrics across the platform in dashboards, monitors, and performance tracing.

Derive new metrics using a combination of existing model dimensions and functions using a SQL-like query language. This allows you to combine your model's features, tags, predictions, and actuals using aggregations, operators, and metric functions.

Use this page to learn:

1. The Arize Query Language

2. How to set up a custom metric

3. How to use your custom metric

Arize Query Language

Custom metrics are defined using a bespoke easy-to-use SQL-like syntax. While the Arize query language and SQL are very similar, the Arize query language deviates from SQL by handling time granularity and time ranges within the Arize platform. This allows you to easily switch time ranges and granularities without changing your query.

Custom Metric Example

This example will compare constructing a custom metric in SQL versus the Arize SQL-like query language.

Using a loan model, we'll calculate the average dollar loss as a percentage of total loan volume. Additionally, we want to filter for loan amounts greater than $1000, and where the state is 'CA'. We want to aggregate the result daily.

For more example use cases, please refer to our examples page here.

SQL:

SELECT
	date_trunc('day', timestamp) AS ts,
	SUM(loan_amount - repayment_amount) / SUM(loan_amount) AS avg_loss
FROM 
	Auto_loans_model
WHERE state = 'CA'
AND loan_amount > 1000
AND timestamp > '2022-01-01' 
AND timestamp < now()
GROUP BY 1 

Arize Query Language:

SELECT
SUM(loan_amount - repayment_amount) / SUM(loan_amount)
FROM model
WHERE state = 'CA'
AND loan_amount > 1000

Expressing the same example with the Arize query language looks similar to SQL but is simplified in comparison by removing time granularity and timestamps. The time parameters are handled automatically for you by the Arize date picker.

Learn more about our custom syntax here.

Navigating to the Custom Metric Editor

Navigate to the custom metric editor in the top navigation bar of any model. Click on 'Custom Metrics' to uncover the editor.

Custom Metric Editor

The custom metric editor allows you to write your queries with syntax validation and auto-completion of dimension names. You will be able to preview your query before saving it.

Name and describe your new metric to adequately convey your new metric's purpose when using Arize collaboratively.

Documentation

Similar to a SQL editor, use the documentation panel to familiarize yourself with the custom syntax via the syntax reference and model schema explorer.

Using A Custom Metric

Custom metrics have the same functionality as any other metric within the Arize platform. Use custom metrics in monitors, dashboards, and performance tracing to ensure your model works as expected.

Dashboards

Use custom metrics in a dashboard to focus your troubleshooting efforts in one place.

Custom Metrics are supported in the 'Time Series' widget. Add a custom metric to the plot by selecting the model for your custom metric. Under 'Metric', select 'Custom Metric', and select the metric you want to add.

Monitors

Monitor custom metrics to get alerted when something goes awry.

From the 'Custom Metrics' page, navigate to the '+ actions' and click 'Create Monitor'.

From the performance metric selector, click on 'Custom Metric' to pick from your list of saved custom metrics. From there, follow the monitor setup workflow.

Performance Tracing

Utilize custom metrics on the performance tracing page to drill into problematic features and slices that affect your metric.

From the 'Performance Tracing' page, click on the metric selector and select 'Custom Metric' to pick from your list of saved custom metrics.