How To Set A Baseline

Choose A Model Baseline
To compare changes, perform analysis, and root cause performance degradations, your model needs a model baseline. A model baseline is a reference data set used to compare your current data with either training, validation, or prior time periods in production.
Default Baseline
Your model's monitors are preset with a baseline defined by a moving time range of your model's production data for 2 weeks delayed by 3 days. 
Configure A Model Baseline
To change your model's baseline to other datasets or time periods, navigate to either: 
The 'Config' tab on the right side of the screen
The 'Datasets' tab in the top navigation bar
From either tab, click on the 'Configure Baseline' button where you will be prompted to pick your baseline from production data or pre-production datasets. 
Configure Your Model Baseline
Production & Pre-production Baselines 
Select parts of your production data using fixed or moving time ranges or choose from different versions of training or validation datasets (pre-production).
Configure Your Model Baseline Using Production Data 
Customize a Baseline 
Here are some considerations that may go into choosing a different baseline:
If you are not expecting changes between training and production: use the training dataset as your baseline
If you are expecting large changes between training and production (e.g. upsampling fraud in training, imbalanced): 
1.use a specific/fixed time range from your production data set, such as the initial model launch period where you're actively monitoring its performance
2.use a validation dataset, this allows for monitoring to start from day 0
If you are monitoring for highly fluctuating changes on a regular time interval (e.g. click-through rate): set a rolling window on your production dataset (e.g. every week, every two weeks)
Custom Drift Monitor Baselines 
Separate from your model baseline, you can customize individual drift monitor baselines to identify changes on a feature level based on a few different settings: 
A different fixed time range: you expect large changes for a specific feature 
A moving time range: identify fluctuating changes for a feature over time
Different versions: compare distribution changes of your current model with versions  
Specific filters: in the case of new, changing, or problematic features, see how different features/tags/etc affect your model's distribution
Edit Monitor Baseline
In your drift monitor of interest, click 'Edit Monitor' from the top right, scroll down to the 'Baseline Distribution' tab, and click 'Use Custom Baseline' to edit advanced settings or change your baseline time window. 
Drift Monitor With Custom Moving Baseline
Drift Monitor With A Custom Filtered Baseline
Last modified 3mo ago