LogoLogo
Python SDKSlack
  • Documentation
  • Cookbooks
  • Self-Hosting
  • Release Notes
  • Reference
  • Arize AI
  • Quickstarts
  • โœจArize Copilot
  • Arize AI for Agents
  • Concepts
    • Agent Evaluation
    • Tracing
      • What is OpenTelemetry?
      • What is OpenInference?
      • Openinference Semantic Conventions
    • Evaluation
  • ๐ŸงชDevelop
    • Quickstart: Experiments
    • Datasets
      • Create a dataset
      • Update a dataset
      • Export a dataset
    • Experiments
      • Run experiments
      • Run experiments with code
        • Experiments SDK differences in AX vs Phoenix
        • Log experiment results via SDK
      • Evaluate experiments
      • Evaluate experiment with code
      • CI/CD with experiments
        • Github Action Basics
        • Gitlab CI/CD Basics
      • Download experiment
    • Prompt Playground
      • Use tool calling
      • Use image inputs
    • Playground Integrations
      • OpenAI
      • Azure OpenAI
      • AWS Bedrock
      • VertexAI
      • Custom LLM Models
    • Prompt Hub
  • ๐Ÿง Evaluate
    • Online Evals
      • Run evaluations in the UI
      • Run evaluations with code
      • Test LLM evaluator in playground
      • View task details & logs
      • โœจCopilot: Eval Builder
      • โœจCopilot: Eval Analysis
      • โœจCopilot: RAG Analysis
    • Experiment Evals
    • LLM as a Judge
      • Custom Eval Templates
      • Arize Templates
        • Agent Tool Calling
        • Agent Tool Selection
        • Agent Parameter Extraction
        • Agent Path Convergence
        • Agent Planning
        • Agent Reflection
        • Hallucinations
        • Q&A on Retrieved Data
        • Summarization
        • Code Generation
        • Toxicity
        • AI vs Human (Groundtruth)
        • Citation
        • User Frustration
        • SQL Generation
    • Code Evaluations
    • Human Annotations
  • ๐Ÿ”ญObserve
    • Quickstart: Tracing
    • Tracing
      • Setup tracing
      • Trace manually
        • Trace inputs and outputs
        • Trace function calls
        • Trace LLM, Retriever and Tool Spans
        • Trace prompt templates & variables
        • Trace as Inferences
        • Send Traces from Phoenix -> Arize
        • Advanced Tracing (OTEL) Examples
      • Add metadata
        • Add events, exceptions and status
        • Add attributes, metadata and tags
        • Send data to a specific project
        • Get the current span context and tracer
      • Configure tracing options
        • Configure OTEL tracer
        • Mask span attributes
        • Redact sensitive data from traces
        • Instrument with OpenInference helpers
      • Query traces
        • Filter Traces
          • Time Filtering
        • Export Traces
        • โœจAI Powered Search & Filter
        • โœจAI Powered Trace Analysis
        • โœจAI Span Analysis & Evaluation
    • Tracing Integrations
      • OpenAI
      • OpenAI Agents SDK
      • LlamaIndex
      • LlamaIndex Workflows
      • LangChain
      • LangGraph
      • Hugging Face smolagents
      • Autogen
      • Google GenAI (Gemini)
      • Model Context Protocol (MCP)
      • Vertex AI
      • Amazon Bedrock
      • Amazon Bedrock Agents
      • MistralAI
      • Anthropic
      • LangFlow
      • Haystack
      • LiteLLM
      • CrewAI
      • Groq
      • DSPy
      • Guardrails AI
      • Prompt flow
      • Vercel AI SDK
      • Llama
      • Together AI
      • OpenTelemetry (arize-otel)
      • BeeAI
    • Evals on Traces
    • Guardrails
    • Sessions
    • Dashboards
      • Dashboard Widgets
      • Tracking Token Usage
      • โœจCopilot: Dashboard Widget Creation
    • Monitors
      • Integrations: Monitors
        • Slack
          • Manual Setup
        • OpsGenie
        • PagerDuty
      • LLM Red Teaming
    • Custom Metrics & Analytics
      • Arize Query Language Syntax
        • Conditionals and Filters
        • All Operators
        • All Functions
      • Custom Metric Examples
      • โœจCopilot: ArizeQL Generator
  • ๐Ÿ“ˆMachine Learning
    • Machine Learning
      • User Guide: ML
      • Quickstart: ML
      • Concepts: ML
        • What Is A Model Schema
        • Delayed Actuals and Tags
        • ML Glossary
      • How To: ML
        • Upload Data to Arize
          • Pandas SDK Example
          • Local File Upload
            • File Upload FAQ
          • Table Ingestion Tuning
          • Wildcard Paths for Cloud Storage
          • Troubleshoot Data Upload
          • Sending Data FAQ
        • Monitors
          • ML Monitor Types
          • Configure Monitors
            • Notifications Providers
          • Programmatically Create Monitors
          • Best Practices for Monitors
        • Dashboards
          • Dashboard Widgets
          • Dashboard Templates
            • Model Performance
            • Pre-Production Performance
            • Feature Analysis
            • Drift
          • Programmatically Create Dashboards
        • Performance Tracing
          • Time Filtering
          • โœจCopilot: Performance Insights
        • Drift Tracing
          • โœจCopilot: Drift Insights
          • Data Distribution Visualization
          • Embeddings for Tabular Data (Multivariate Drift)
        • Custom Metrics
          • Arize Query Language Syntax
            • Conditionals and Filters
            • All Operators
            • All Functions
          • Custom Metric Examples
          • Custom Metrics Query Language
          • โœจCopilot: ArizeQL Generator
        • Troubleshoot Data Quality
          • โœจCopilot: Data Quality Insights
        • Explainability
          • Interpreting & Analyzing Feature Importance Values
          • SHAP
          • Surrogate Model
          • Explainability FAQ
          • Model Explainability
        • Bias Tracing (Fairness)
        • Export Data to Notebook
        • Automate Model Retraining
        • ML FAQ
      • Use Cases: ML
        • Binary Classification
          • Fraud
          • Insurance
        • Multi-Class Classification
        • Regression
          • Lending
          • Customer Lifetime Value
          • Click-Through Rate
        • Timeseries Forecasting
          • Demand Forecasting
          • Churn Forecasting
        • Ranking
          • Collaborative Filtering
          • Search Ranking
        • Natural Language Processing (NLP)
        • Common Industry Use Cases
      • Integrations: ML
        • Google BigQuery
          • GBQ Views
          • Google BigQuery FAQ
        • Snowflake
          • Snowflake Permissions Configuration
        • Databricks
        • Google Cloud Storage (GCS)
        • Azure Blob Storage
        • AWS S3
          • Private Image Link Access Via AWS S3
        • Kafka
        • Airflow Retrain
        • Amazon EventBridge Retrain
        • MLOps Partners
          • Algorithmia
          • Anyscale
          • Azure & Databricks
          • BentoML
          • CML (DVC)
          • Deepnote
          • Feast
          • Google Cloud ML
          • Hugging Face
          • LangChain ๐Ÿฆœ๐Ÿ”—
          • MLflow
          • Neptune
          • Paperspace
          • PySpark
          • Ray Serve (Anyscale)
          • SageMaker
            • Batch
            • RealTime
            • Notebook Instance with Greater than 20GB of Data
          • Spell
          • UbiOps
          • Weights & Biases
      • API Reference: ML
        • Python SDK
          • Pandas Batch Logging
            • Client
            • log
            • Schema
            • TypedColumns
            • EmbeddingColumnNames
            • ObjectDetectionColumnNames
            • PromptTemplateColumnNames
            • LLMConfigColumnNames
            • LLMRunMetadataColumnNames
            • NLP_Metrics
            • AutoEmbeddings
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
          • Single Record Logging
            • Client
            • log
            • TypedValue
            • Ranking
            • Multi-Class
            • Object Detection
            • Embedding
            • LLMRunMetadata
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
        • Java SDK
          • Constructor
          • log
          • bulkLog
          • logValidationRecords
          • logTrainingRecords
        • R SDK
          • Client$new()
          • Client$log()
        • Rest API
    • Computer Vision
      • How to: CV
        • Generate Embeddings
          • How to Generate Your Own Embedding
          • Let Arize Generate Your Embeddings
        • Embedding & Cluster Analyzer
        • โœจCopilot: Embedding Summarization
        • Similarity Search
        • Embedding Drift
        • Embeddings FAQ
      • Integrations: CV
      • Use Cases: CV
        • Image Classification
        • Image Segmentation
        • Object Detection
      • API Reference: CV
Powered by GitBook

Support

  • Chat Us On Slack
  • support@arize.com

Get Started

  • Signup For Free
  • Book A Demo

Copyright ยฉ 2025 Arize AI, Inc

On this page
  • Ranking Model Overview
  • General Ranking Model Schema
  • Case 1: Ranking with Relevance Score
  • Case 2: Ranking with Single Label
  • Case 3: Ranking with Multiple Labels
  • Ranking Single or Multi-Label + AUC and LogLoss
  • Ranking Performance Metrics
  • NDCG @k
  • Selecting Relevance Score or Label - Attribution Model
  • Ranking Quick Definitions

Was this helpful?

  1. Machine Learning
  2. Machine Learning
  3. Use Cases: ML

Ranking

How to log your model schema for ranking models

Last updated 6 months ago

Was this helpful?

Ranking Model Overview

There are four key ranking model use cases to consider:

  • Search Ranking

  • Collaborative Filtering Recommender Systems

  • Content Filtering Recommender Systems

  • Classification-Based Ranking Models

Different metrics are used for ranking model evaluation based on your model use case, score, and label availability. The case determines the available performance metrics. Click for all valid model types and metric combinations.

Ranking Cases
Example Use Case
Expected Fields
Performance Metrics

Model predicts score used to rank

rank, relevance score

Model predicts binary actions a user can take which is used to rank

rank, relevance_labels

Model predicts multiple actions a user can take which is used to rank

rank, relevance_labels (list of strings)

Model can also be evaluated using AUC + LogLoss

Ranking Case 2 or 3, prediction score

General Ranking Model Schema

Ranking models have a few unique model schema fields that help Arize effectively monitor, trace, and visualize your ranking model data.

  • Prediction Group ID: A subgroup of prediction data. Max 100 ranked items within each group

  • Rank: Unique value within each prediction group (1-100)

  • Relevancy Score/Label: Ground truth score/label associated with the model

Case 1: Ranking with Relevance Score

In the ranking model context, a relevance score is a ground truth numerical score where the higher the relevance_score, the more important the item is. For example, if an item was clicked on it may have a relevance score of 0.5, whereas if it was purchased that relevancy score would be 1.

rank and relevance_score are required to compute rank-aware evaluation metrics on your model.

Ranking Model Fields
Data Type
Example

rank

int from 1-100

1

relevance_score

numeric (float | int)

0.5

prediction_group_id

string limited to 128 characters

148

Code Example

Example Row

state
price
search_id
rank
relevance_score
prediction_ts

ca

98

148

1

0.5

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_group_id_column_name = "search_id",
    rank_column_name = "rank",
    relevance_score_column_name = "relevance_score",
    feature_column_names=["state", "price"],
    tag_column_names=tag_columns,
)

response = arize_client.log(
    dataframe=df,
    model_id="ranking-relevance-score-batch-ingestion-tutorial",
    model_version="1.0",
    model_type=ModelTypes.RANKING,
    metrics_validation=[Metrics.RANKING],
    environment=Environments.PRODUCTION,
    schema=schema,
)

For more details on Python Batch API Reference, visit here:

# import extra dependencies
from arize.utils.types import Environments, ModelTypes, Schema, RankingPredictionLabel, RankingActualLabel

# define prediction label arguments
pred_label = RankingPredictionLabel(
    group_id="148", 
    rank=1, 
    score=0.155441 
)

# define actual label argument
act_label = RankingActualLabel(
    relevance_score=0.5
)

# log data to Arize
response = arize_client.log(
    model_id="demo-ranking-with-relevance-score",
    model_version="v1",
    environment=Environments.PRODUCTION,
    model_type=ModelTypes.RANKING,
    prediction_id="311103e3-a493-40ea-a21a-e457d617c956",
    prediction_label=pred_label,
    actual_label=act_label,
    features=features
)

Learn how to upload files via various Data Connectors:

Case 2: Ranking with Single Label

Ranking Model Fields
Data Type
Example

rank

int from 1-100

1

relevance_labels

string

โ€œclickโ€

prediction_group_id

string limited to 128 characters

148

Code Example

Example Row

state
price
search_id
rank
actual_relevancy
prediction_ts

ca

98

148

1

"not relevant"

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_group_id_column_name = "search_id",
    rank_column_name = "rank",
    relevance_labels_column_name = "actual_relevancy",
    feature_column_names=["state", "price"],
    tag_column_names=tag_columns,
)

response = arize_client.log(
    dataframe=df,
    model_id="ranking-single-label-batch-ingestion-tutorial",
    model_version="1.0",
    model_type=ModelTypes.RANKING,
    metrics_validation=[Metrics.RANKING, Metrics.RANKING_LABEL],
    environment=Environments.PRODUCTION,
    schema=schema,
)

For more details on Python Batch API Reference, visit here:

# import extra dependencies
from arize.utils.types import Environments, ModelTypes, Schema, RankingPredictionLabel, RankingActualLabel

# define prediction label arguments
pred_label = RankingPredictionLabel(
    group_id="148", 
    rank=1, 
    label="relevant" 
)

# define actual label argument
act_label = RankingActualLabel(
    relevance_labels=["Not relevant"]
)

# log data to Arize
response = arize_client.log(
    model_id="demo-ranking-with-single-label",
    model_version="v1",
    environment=Environments.PRODUCTION,
    model_type=ModelTypes.RANKING,
    prediction_id="311103e3-a493-40ea-a21a-e457d617c956",
    prediction_label=pred_label,
    actual_label=act_label,
    features=features
)

Learn how to upload files via various Data Connectors:

Case 3: Ranking with Multiple Labels

Since ground truth can contain multiple events, you can pass in multiple ground truth labels in a list.

Ranking Model Fields
Data Type
Example

rank

int from 1-100

1

relevance_labels

list of strings

[โ€œclickโ€, โ€œfavoriteโ€, โ€œbuyโ€]

prediction_group_id

string limited to 128 characters

148

Code Example

Example Row

state
price
search_id
rank
attributions
prediction_ts

ca

98

148

1

"click, favorite, buy"

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_group_id_column_name = "search_id",
    rank_column_name = "rank",
    relevance_labels_column_name = "attributions"
    feature_column_names=["state", "price"],
    tag_column_names=tag_columns,
)

response = arize_client.log(
    dataframe=df,
    model_id="ranking-multiple-labels-batch-ingestion-tutorial",
    model_version="1.0",
    model_type=ModelTypes.RANKING,
    metrics_validation=[Metrics.RANKING, Metrics.RANKING_LABEL],
    environment=Environments.PRODUCTION,
    schema=schema,
)

For more details on Python Batch API Reference, visit here:

# import extra dependencies
from arize.utils.types import Environments, ModelTypes, Schema, RankingPredictionLabel, RankingActualLabel

# define prediction label arguments
pred_label = RankingPredictionLabel(
    group_id="148", 
    rank=2, 
    label="click" 
)

# define actual label argument
act_label = RankingActualLabel(
    relevance_labels=["book", "click"]
)

# log data to Arize
response = arize_client.log(
    model_id="demo-ranking-with-multiple-labels",
    model_version="v1",
    environment=Environments.PRODUCTION,
    model_type=ModelTypes.RANKING,
    prediction_id="dd19bee3-e7f4-4207-aef9-3abdad2a9be0",
    prediction_label=pred_label,
    actual_label=act_label,
    features=features
)

Learn how to upload files via various Data Connectors:

Ranking Single or Multi-Label + AUC and LogLoss

AUC and LogLoss are computed based on prediction_score and relevance_labels (or default relevance_labels in the case of multi-label).

Ranking Model Fields
Data Type
Example

rank

int from 1-100

1

prediction_score

float

0.5

prediction_group_id

string limited to 128 characters

148

Ranking Performance Metrics

Rank-aware evaluation metrics: NDCG @k (MAP @K & MRR coming soon)

Evaluation metrics: AUC, PR-AUC, LogLoss

NDCG @k

What is @k?

The k value determines the metric computation up to position k in a list.

Selecting Relevance Score or Label - Attribution Model

A relevance score is required to calculate rank-aware evaluation metrics. If your relevance_score is unavailable, the Arize platform will calculate a relevance_score using a simple attribution model with a prediction label and a relevance label. Arize computes a binary relevance value (0/1) based on the default positive class.

  • Positive class "buy" and relevance label is "buy" --> relevance will be attributed to 1.

  • Positive class "buy" and relevance label is else --> relevance will be attributed to 0.

  • Positive class "buy" and relevance labels are ["buy", "click", "scroll"] --> relevance will be attributed to sum([1,0,0])

Ranking Quick Definitions

Ranking model: Assigns a rank to each item in a prediction group (also known as a batch or query), across many possible groups.

Prediction Group: A group of predictions within which items are ranked.

Example: A user of a hotel booking site types in a search term (โ€œskiingโ€) and is presented with a list of results representing a single query

Rank: The predicted rank of an item in a prediction group (Integer between 1-100).

Example: Each item in the search prediction group has a rank determined by the model (i.e. Aspen is assigned rank=1, Tahoe is assigned rank=2, etc. based on input features and query features to the model)

Relevance Score (i.e. Actual Scores): The ground truth relevance score (numeric). Higher scores denote higher relevance.

Example: Each item in the search prediction group has a score determined by the action a user took on the item (i.e. โ€œclickingโ€ on an item indicates relevance score = 0.5, purchasing an item indicates relevance score = 1)

Rank-Aware Evaluation Metric: A rank-aware evaluation metric is an evaluation metric that gauges rank order and relevancy of predictions.

Rank-aware evaluation metrics include NDCG, MRR, and MAP. Note that MRR and MAP also require relevance_labels to be provided to be computed.

, , ,

, , ,

, , , , AUC, PR-AUC,

Download an example Parquet file: Open parquet reader.

In this case, relevance_scoredoes not need to be passed in. Since relevance_score is required to compute rank-aware evaluation metrics, Arize uses an attribution model to create a relevance_score based on your positive class and relevance_labels. Learn more about our attribution model .

Download an example Parquet file: Open parquet reader.

In this case relevance_score does not need to be passed in. Since relevance_score is required to compute rank-aware evaluation metrics, Arize uses an attribution model to create a relevance_score based on your positive class and relevance_labels. Learn more about our attribution model .

Download an example Parquet file: Open parquet reader.

Normalized discounted cumulative gain (NDCG) is a that measures a model's ability to rank query results in the order of the highest relevance (graded relevance). You can read more about how NDCG is computed .

Arize supports ,, and ranking models

๐Ÿ“ˆ
1618590882
1618590882
1618590882
Pandas Batch Logging
here
Google Cloud Storage (GCS)
AWS S3
Azure Blob Storage
Google BigQuery
Pandas Batch Logging
here
Google Cloud Storage (GCS)
AWS S3
Azure Blob Storage
Google BigQuery
Pandas Batch Logging
here
Google Cloud Storage (GCS)
AWS S3
Azure Blob Storage
Google BigQuery
here
here
here
rank-aware evaluation metric
Case 1: Ranking with Relevance Score
NDCG
Case 2: Ranking with Single Label
GroupAUC
MAP
MRR
NDCG
Case 3: Ranking with Multiple Labels
GroupAUC
MAP
MRR
NDCG
Ranking + AUC and LogLoss
GroupAUC
MAP
MRR
Log Loss
NDCG
https://storage.cloud.google.com/arize-assets/documentation-sample-data/data-ingestion/ranking-assets/ranking-relevance-score-sample-data.parquet?authuser=0storage.cloud.google.com
pointwise
pairwise
listwise
https://storage.cloud.google.com/arize-assets/documentation-sample-data/data-ingestion/ranking-assets/ranking-single-label-sample-data.parquet?authuser=0storage.cloud.google.com
https://storage.cloud.google.com/arize-assets/documentation-sample-data/data-ingestion/ranking-assets/ranking-multiple-labels-sample-data.parquet?authuser=0storage.cloud.google.com
here
Google Colaboratory
Google Colaboratory
Google Colaboratory
Google Colaboratory
Google Colaboratory
Google Colaboratory
Logo
Logo
Logo
Logo
Logo
Logo