Arize AI
Search…
Ranking

Ranking Models

Ranking models are used by search engines or other information retrieval systems to display query results ranked in the order of the highest relevance. To assess the performance of ranking models, Arize supports logging the sequence of actual relevance scores in the order of the displayed results.

Example Schema

Field
Data Type
Example
Arize Field
Prediction Group ID (often referred to as query or session ID)
string
prediction_group_id (query_id)
prediction_group_id
Prediction Score (required)
List of Floats
rank_score
prediction_score
Rank/Position
List of Ints (must range between 1-100)
rank
rank
Actual Label (optional; it's required if actual scores are not logged)
List of Strings
event/action (e.g buy, click, save, etc.)
actual_lable
Actual Score (optional; it's required if actual labels are not logged)
List of Relevances (floats)
relevance
actual_score

Example ML Use Case

Ranking Models
  • Prediction Label: Set to "relevant" (since only relevant results are displayed).
  • Actual Label: Set to "relevant" or "not relevant" for the first search result, and similarly for each search result if also logging additional results of the same query.
  • Actual Numeric Sequence: Set to a sequence of actual relevance scores, e.g [0, 1, 0, 1], on the first query result. If additional results of the same query are also logged, it should be set to None.

Performance Metrics Supported

If you include the sequence of actual relevance scores, the following metric is supported.
  • NDCG
  • Log Loss
  • AUC

Code Example

This example uses the pandas logger. relevance_scores is the name of a pandas DataFrame column where each row is a list of numbers (or None if a list is not applicable). See this Colab for details.
schema = Schema(
prediction_id_column_name="prediction_id",
...
prediction_group_id_column_name="query_id",
rank="position",
actual_label_column_name="actions_taken",
actual_score_column_name="actual_relevance_scores",
)
response = arize_client.log(
model_id="sample-model-1",
model_version= "v1",
model_type=ModelTypes.SCORE_CATEGORICAL,
environment=Environments.PRODUCTION,
dataframe=test_dataframe,
schema=schema,
)
Copy link
Outline
Ranking Models
Example Schema
Example ML Use Case
Performance Metrics Supported
Code Example