LogoLogo
Python SDKSlack
  • Documentation
  • Cookbooks
  • Self-Hosting
  • Release Notes
  • Reference
  • Arize AI
  • Quickstarts
  • โœจArize Copilot
  • Arize AI for Agents
  • Concepts
    • Agent Evaluation
    • Tracing
      • What is OpenTelemetry?
      • What is OpenInference?
      • Openinference Semantic Conventions
    • Evaluation
  • ๐ŸงชDevelop
    • Quickstart: Experiments
    • Datasets
      • Create a dataset
      • Update a dataset
      • Export a dataset
    • Experiments
      • Run experiments
      • Run experiments with code
        • Experiments SDK differences in AX vs Phoenix
        • Log experiment results via SDK
      • Evaluate experiments
      • Evaluate experiment with code
      • CI/CD with experiments
        • Github Action Basics
        • Gitlab CI/CD Basics
      • Download experiment
    • Prompt Playground
      • Use tool calling
      • Use image inputs
    • Playground Integrations
      • OpenAI
      • Azure OpenAI
      • AWS Bedrock
      • VertexAI
      • Custom LLM Models
    • Prompt Hub
  • ๐Ÿง Evaluate
    • Online Evals
      • Run evaluations in the UI
      • Run evaluations with code
      • Test LLM evaluator in playground
      • View task details & logs
      • โœจCopilot: Eval Builder
      • โœจCopilot: Eval Analysis
      • โœจCopilot: RAG Analysis
    • Experiment Evals
    • LLM as a Judge
      • Custom Eval Templates
      • Arize Templates
        • Agent Tool Calling
        • Agent Tool Selection
        • Agent Parameter Extraction
        • Agent Path Convergence
        • Agent Planning
        • Agent Reflection
        • Hallucinations
        • Q&A on Retrieved Data
        • Summarization
        • Code Generation
        • Toxicity
        • AI vs Human (Groundtruth)
        • Citation
        • User Frustration
        • SQL Generation
    • Code Evaluations
    • Human Annotations
  • ๐Ÿ”ญObserve
    • Quickstart: Tracing
    • Tracing
      • Setup tracing
      • Trace manually
        • Trace inputs and outputs
        • Trace function calls
        • Trace LLM, Retriever and Tool Spans
        • Trace prompt templates & variables
        • Trace as Inferences
        • Send Traces from Phoenix -> Arize
        • Advanced Tracing (OTEL) Examples
      • Add metadata
        • Add events, exceptions and status
        • Add attributes, metadata and tags
        • Send data to a specific project
        • Get the current span context and tracer
      • Configure tracing options
        • Configure OTEL tracer
        • Mask span attributes
        • Redact sensitive data from traces
        • Instrument with OpenInference helpers
      • Query traces
        • Filter Traces
          • Time Filtering
        • Export Traces
        • โœจAI Powered Search & Filter
        • โœจAI Powered Trace Analysis
        • โœจAI Span Analysis & Evaluation
    • Tracing Integrations
      • OpenAI
      • OpenAI Agents SDK
      • LlamaIndex
      • LlamaIndex Workflows
      • LangChain
      • LangGraph
      • Hugging Face smolagents
      • Autogen
      • Google GenAI (Gemini)
      • Model Context Protocol (MCP)
      • Vertex AI
      • Amazon Bedrock
      • Amazon Bedrock Agents
      • MistralAI
      • Anthropic
      • LangFlow
      • Haystack
      • LiteLLM
      • CrewAI
      • Groq
      • DSPy
      • Guardrails AI
      • Prompt flow
      • Vercel AI SDK
      • Llama
      • Together AI
      • OpenTelemetry (arize-otel)
      • BeeAI
    • Evals on Traces
    • Guardrails
    • Sessions
    • Dashboards
      • Dashboard Widgets
      • Tracking Token Usage
      • โœจCopilot: Dashboard Widget Creation
    • Monitors
      • Integrations: Monitors
        • Slack
          • Manual Setup
        • OpsGenie
        • PagerDuty
      • LLM Red Teaming
    • Custom Metrics & Analytics
      • Arize Query Language Syntax
        • Conditionals and Filters
        • All Operators
        • All Functions
      • Custom Metric Examples
      • โœจCopilot: ArizeQL Generator
  • ๐Ÿ“ˆMachine Learning
    • Machine Learning
      • User Guide: ML
      • Quickstart: ML
      • Concepts: ML
        • What Is A Model Schema
        • Delayed Actuals and Tags
        • ML Glossary
      • How To: ML
        • Upload Data to Arize
          • Pandas SDK Example
          • Local File Upload
            • File Upload FAQ
          • Table Ingestion Tuning
          • Wildcard Paths for Cloud Storage
          • Troubleshoot Data Upload
          • Sending Data FAQ
        • Monitors
          • ML Monitor Types
          • Configure Monitors
            • Notifications Providers
          • Programmatically Create Monitors
          • Best Practices for Monitors
        • Dashboards
          • Dashboard Widgets
          • Dashboard Templates
            • Model Performance
            • Pre-Production Performance
            • Feature Analysis
            • Drift
          • Programmatically Create Dashboards
        • Performance Tracing
          • Time Filtering
          • โœจCopilot: Performance Insights
        • Drift Tracing
          • โœจCopilot: Drift Insights
          • Data Distribution Visualization
          • Embeddings for Tabular Data (Multivariate Drift)
        • Custom Metrics
          • Arize Query Language Syntax
            • Conditionals and Filters
            • All Operators
            • All Functions
          • Custom Metric Examples
          • Custom Metrics Query Language
          • โœจCopilot: ArizeQL Generator
        • Troubleshoot Data Quality
          • โœจCopilot: Data Quality Insights
        • Explainability
          • Interpreting & Analyzing Feature Importance Values
          • SHAP
          • Surrogate Model
          • Explainability FAQ
          • Model Explainability
        • Bias Tracing (Fairness)
        • Export Data to Notebook
        • Automate Model Retraining
        • ML FAQ
      • Use Cases: ML
        • Binary Classification
          • Fraud
          • Insurance
        • Multi-Class Classification
        • Regression
          • Lending
          • Customer Lifetime Value
          • Click-Through Rate
        • Timeseries Forecasting
          • Demand Forecasting
          • Churn Forecasting
        • Ranking
          • Collaborative Filtering
          • Search Ranking
        • Natural Language Processing (NLP)
        • Common Industry Use Cases
      • Integrations: ML
        • Google BigQuery
          • GBQ Views
          • Google BigQuery FAQ
        • Snowflake
          • Snowflake Permissions Configuration
        • Databricks
        • Google Cloud Storage (GCS)
        • Azure Blob Storage
        • AWS S3
          • Private Image Link Access Via AWS S3
        • Kafka
        • Airflow Retrain
        • Amazon EventBridge Retrain
        • MLOps Partners
          • Algorithmia
          • Anyscale
          • Azure & Databricks
          • BentoML
          • CML (DVC)
          • Deepnote
          • Feast
          • Google Cloud ML
          • Hugging Face
          • LangChain ๐Ÿฆœ๐Ÿ”—
          • MLflow
          • Neptune
          • Paperspace
          • PySpark
          • Ray Serve (Anyscale)
          • SageMaker
            • Batch
            • RealTime
            • Notebook Instance with Greater than 20GB of Data
          • Spell
          • UbiOps
          • Weights & Biases
      • API Reference: ML
        • Python SDK
          • Pandas Batch Logging
            • Client
            • log
            • Schema
            • TypedColumns
            • EmbeddingColumnNames
            • ObjectDetectionColumnNames
            • PromptTemplateColumnNames
            • LLMConfigColumnNames
            • LLMRunMetadataColumnNames
            • NLP_Metrics
            • AutoEmbeddings
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
          • Single Record Logging
            • Client
            • log
            • TypedValue
            • Ranking
            • Multi-Class
            • Object Detection
            • Embedding
            • LLMRunMetadata
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
        • Java SDK
          • Constructor
          • log
          • bulkLog
          • logValidationRecords
          • logTrainingRecords
        • R SDK
          • Client$new()
          • Client$log()
        • Rest API
    • Computer Vision
      • How to: CV
        • Generate Embeddings
          • How to Generate Your Own Embedding
          • Let Arize Generate Your Embeddings
        • Embedding & Cluster Analyzer
        • โœจCopilot: Embedding Summarization
        • Similarity Search
        • Embedding Drift
        • Embeddings FAQ
      • Integrations: CV
      • Use Cases: CV
        • Image Classification
        • Image Segmentation
        • Object Detection
      • API Reference: CV
Powered by GitBook

Support

  • Chat Us On Slack
  • support@arize.com

Get Started

  • Signup For Free
  • Book A Demo

Copyright ยฉ 2025 Arize AI, Inc

On this page
  • Example Custom Metrics
  • Business metrics
  • Percent of Features With Value > X
  • Weighted Average Performance Metric
  • Count Distinct
  • Performance metrics
  • Precision
  • Percent Error of Model > X
  • Choosing a different threshold for classification metrics
  • F Beta
  • Pinball Loss Function
  • MAX_PRECISION
  • Data Consistency
  • Match Rate
  • Average Residual Error

Was this helpful?

  1. Machine Learning
  2. Machine Learning
  3. How To: ML
  4. Custom Metrics

Custom Metric Examples

Common example use cases

Last updated 6 months ago

Was this helpful?

Example Custom Metrics

Custom metrics are an extremely powerful tool to evaluate many dimensions of your ML model. From analyzing the business impact of your model to calculating a moving average, you can leverage custom metrics in many ways. Use this page as a guide on how you can create custom metrics tailored to your ML needs.

Business metrics

The following examples use the arize-demo-fraud-use-case model to calculate business KPIs and other useful statistics using custom metrics.

Percent of Features With Value > X

We'll use this example to calculate the percentage of predictions with a fico score below 600. Assume that the lending model automatically rejects customers with scores below 600, so you want to track this metric to determine the health of inbound applications. We can do this by using a FILTER (WHERE ... ) clause, which allows us to apply the < 600 filter to only the numerator and not the denominator.

SELECT
COUNT(*) FILTER (WHERE fico_score < 600) / COUNT(*) * 100
FROM model

Learn more about FILTER (WHERE) clauses

Weighted Average Performance Metric

Weighted averages can be useful when different predictions are important to your business.

To do this, we'll calculate recall but weigh the predictions by the loan amount. This way, a false negative on more expensive loans will have a higher impact on the score than a false negative on a cheaper loan.

We can use a WHERE clause to filter for the actual fraud cases, and a FILTER (WHERE ... ) clause to filter the numerator where the model incorrectly predicted not_fraud .

SELECT

SUM(loan_amount) FILTER (WHERE categoricalPredictionLabel = 'fraud') / 
SUM(loan_amount) 

FROM model
WHERE categoricalActualLabel = 'fraud'

Count Distinct

You can use APPROX_COUNT_DISTINCT to get the cardinality of a dimension. In this example, we can calculate the average total loan amount per merchant.

SELECT 

SUM(loan_amount) / APPROX_COUNT_DISTINCT(merchant_ID)

FROM model

Performance metrics

Use natively supported performance metrics as a function that can take multiple arguments for enhanced performance monitoring flexibility. Additionally, create completely new metrics using conditionals and other logic shown below.

Precision

PRECISION ()

When a model has multiple predictions that are sent as features or tags, the PRECISION function allows you to specify the exact columns used. In this example, a user may have a model with multiple predictions, each one sent in as a tag or feature. The PRECISION() function allows you to specify different columns to use for predictions or actuals, using the predicted and actual keyword arguments:

SELECT
PRECISION(
    predicted=multi_output_prediction, 
    actual=multi_output_actual, 
    pos_class='custom_positive_class'
)
FROM model

Percent Error of Model > X

Calculate the percent error to compare your model's predictions over actual values. In this example, we'll create a custom metric to calculate the percent error greater than 9%.

select avg(
    case 
        when abs(scorePredictionLabel - scoreActualLabel)/scoreActualLabel > 9 
        then 1 
        else 0 
    end) 
from model 
where scorePredictionLabel is not null
    and scoreActualLabel is not null 
    and scoreActualLabel != 0

Choosing a different threshold for classification metrics

By combining classification metrics with case statements, users can use a different classification threshold than the currently ingested data.

To do this, you can use the predicted argument of the metric function. This expects the prediction label, which you can conditionally set using a CASE statement.

SELECT
PRECISION(
    predicted=(CASE WHEN scorePredictionLabel > 0.25 THEN 'fraud' ELSE 'not_fraud' END),
    pos_class='fraud'
)
FROM model

F Beta

F1 score is a special case of F beta, where precision and recall are equally weighted. In some use cases, you may prefer to weigh precision more than recall, or vice versa, based on business needs. The F_BETA function allows you to specify the beta parameter to adjust this weighting. In this example, we want to weigh precision twice as much as recall, with a beta score of 0.5.

SELECT 
F_BETA(beta=0.5)
FROM model

Pinball Loss Function

Custom metrics can be used to create performance metrics such as pinball loss. Pinball loss is useful for quantile forecasts. An example could be a weather prediction, such as an 80% chance of 20 degrees or colder weather.

Let y be the actuals and z be the prediction. ฮฑ is the quantile, in this case, 0.8 (for 80% chance). The formula for pinball loss is as such.

L(y, z) = (y - z) * ฮฑ       if y >= z
        = (z - y) * (1 - ฮฑ) if z > y

This is what pinball loss would look like for a numeric model.

SELECT
AVG(
    CASE 
        WHEN numericActualLabel >= numericPredictionLabel 
        THEN (numericActualLabel - numericPredictionLabel) * 0.8
    ELSE (numericPredictionLabel - numericActualLabel) * (1 - 0.8)
    END
)
FROM model

Custom metrics are not just limited to predictions and actuals. For example, if multiple predictions at multiple quantiles are sent in as tags, such as a tag prediction_temp_p80 for the 80th percentile temperature prediction, pinball loss could look like this:

SELECT
AVG(
    CASE 
        WHEN numericActualLabel >= prediction_temp_p80 
        THEN (numericActualLabel - prediction_temp_p80) * 0.8
    ELSE (numericActualLabel - prediction_temp_p80) * (1 - 0.8)
    END
)
FROM model

MAX_PRECISION

Calculate Max Precision with custom threshold using case when statements

select 
MAX_PRECISION(pos_class='fraud',
predicted = (CASE WHEN "scorePredictionLabel" > 0.7 THEN 'fraud' ELSE 'not_fraud' END),
actual = "categoricalActualLabel",
group_by_column="merchant_ID") 
from model

Data Consistency

Data consistency measures the discrepancies between two datasets (i.e. online versus offline features). To compare the same set of features, we recommend sending in one dataset as feature values and the other dataset as tag values.

Match Rate

Percent of unmatched feature values for a given column.

SELECT COUNT(*) FILTER(WHERE (online_feature != offline_feature)) / COUNT(*) * 100
FROM model

Average Residual Error

The average difference between two datasets for a given column.

SELECT AVG(ABS(offline_feature - online_feature))
FROM model

Learn more about APPROX_COUNT_DISTINCT and other aggregation functions

View the documentation for our performance metrics

Learn more about PRECISION and related functions

This works for all with the predicted argument.

Learn more about F_BETA and related functions

Learn more about CASE statements and other conditional structures

For support with creating data consistency metrics, reach out to support@arize.com or message us on Slack .

๐Ÿ“ˆ
here
here.
here.
here.
classification metrics
here.
here.
here.