How to log your model schema for ranking models
To assess the ranking model performance, Arize supports logging the relevance scores or the sequence of actual labels in the order of the displayed results.
This metric is supported if you include actual relevance scores and rank to your model schema
MAP @ k, MRR, LogLoss, and AUC are also supported using the optional parameters in the table below.
Arize supports pointwise, pairwise and listwise ranking models. Depending on the ranking model approach
prediction_scorewill be used.
actual_scoreis unavailable, you must send either
prediction_scoreto compute the following metrics:
- prediction_score : Log Loss, AUC, PR-AUC
# Declare the schema of the dataframe you're sending (feature columns, predictions, timestamp, actuals)
schema = Schema(
prediction_group_id_column_name = "srch_id",
rank_column_name = "rank",
actual_score_column_name = "ActualRelevancyScore",
# Log the dataframe with the schema mapping
response = arize_client.log(
Examples: music playlists, products to advertise, items to include in a subscription box, hotels to promote, etc.
Actual_score (i.e. relevance score) is needed to compute NDCG.
If your relevance score is not available, the Arize platform will calculate a relevance score for you using a simple attribution model with prediction label and actual label. Arize computes a binary relevance value (0/1) based on the default positive class.
Positive class "buy" and receive a "buy" relevance will be attributed to 1.
Positive class "buy" and receive anything else, relevance will be attributed to 0.
Positive class "buy" and receive ["buy", "click", "scroll"], relevance will be attributed to
See more information on this below in Optional Parameters.
What is @k?
The k value determines the metric computation up to position k in a list, for example in the formula for NDCG below.
NDCG is the quotient of DCG and IDCG @ k
Navigate to the 'Config' page anywhere from your Ranking model to edit your K value, and config positive class if "Actual Label" is used for NDCG.