Arize AI
Search…
⌃K

Ranking

How to log your model schema for ranking models

Ranking Model Overview

Ranking models are used by search engines or other information retrieval systems to display query results ranked in the order of the highest relevance.
To assess the ranking model performance, Arize supports logging the relevance scores or the sequence of actual labels in the order of the displayed results.

Performance Metrics

[email protected] (normalized discounted cumulative gain)
This metric is supported if you include actual relevance scores and rank to your model schema
MAP @ k, MRR, LogLoss, and AUC are also supported using the optional parameters in the table below.

Examples

File Type
Link
Python Batch

Ranking Model Schema Required Parameters

Arize Field
Data Type
Example
str
"prediction_id"
prediction_group_id
str
"groupid_123"
timestamp
str
"prediction_ts"
List[str]
['prop_log_historical_price','price_usd','promotion_flag']
model_id
str
'sample-model-1'
str
'v1'
model_type
str
ModelTpes.RANKING
str
Environments.PRODUCTION
rank
int from 1-100
1
actual_score (i.e. relevance score)
float | int
0.6

Ranking Model Schema Optional Parameters

Arize supports pointwise, pairwise and listwise ranking models. Depending on the ranking model approach prediction_label or prediction_score will be used.
If your actual_score is unavailable, you must send either prediction_label & actual_label OR prediction_score to compute the following metrics:
  • prediction_label + actual label : [email protected]k , MRR
    • Note: positive class must be configured in the model config UI as shown below
  • prediction_score : Log Loss, AUC, PR-AUC
Arize Field
Data Type
Example
(optional) prediction_label
string
"buy"
(optional)
actual_label (i.e. attributions)
single string or list of strings
"buy" or ["buy","click","save"]
(optional)
prediction_score
float
1.1

Code Example

# Declare the schema of the dataframe you're sending (feature columns, predictions, timestamp, actuals)
schema = Schema(
prediction_id_column_name="prediction_id",
timestamp_column_name="prediction_ts",
prediction_group_id_column_name = "srch_id",
rank_column_name = "rank",
actual_score_column_name = "ActualRelevancyScore",
feature_column_names=['prop_log_historical_price','price_usd','promotion_flag']
)
# Log the dataframe with the schema mapping
response = arize_client.log(
model_id="sample-model-1",
model_version= "v1",
model_type=ModelTypes.ModelTypes.RANKING,
environment=Environments.PRODUCTION,
dataframe=test_dataframe,
schema=schema
)
This example is for the python batch ingestion method, for other languages, please refer to our examples.
Examples: music playlists, products to advertise, items to include in a subscription box, hotels to promote, etc.

Selecting relevance score or label - Attribution Model

Actual_score (i.e. relevance score) is needed to compute NDCG.
If your relevance score is not available, the Arize platform will calculate a relevance score for you using a simple attribution model with prediction label and actual label. Arize computes a binary relevance value (0/1) based on the default positive class.
Positive class "buy" and receive a "buy" relevance will be attributed to 1.
Positive class "buy" and receive anything else, relevance will be attributed to 0.
Positive class "buy" and receive ["buy", "click", "scroll"], relevance will be attributed to sum([1,0,0])
See more information on this below in Optional Parameters.

Additional Ranking Schema Definitions

Prediction Group ID
Rank
Prediction Label
Prediction Score
Actual Label
Relevance Scores

Performance Metric: NDCG

Normalized discounted cumulative gain (NDCG) is a rank-aware evaluation metric that measures a model's ability to rank query results in the order of the highest relevance (graded relevance). You can read more about how NDCG is computed here, or the implementation here.
What is @k?
The k value determines the metric computation up to position k in a list, for example in the formula for NDCG below.
NDCG is the quotient of DCG and IDCG @ k
Navigate to the 'Config' page anywhere from your Ranking model to edit your K value, and config positive class if "Actual Label" is used for NDCG.