Performance Monitors
Real-time model performance can be tracked across any evaluation dimensions in the Arize platform. Additionally, performance monitors provide granularity down to the hourly level, enabling deep troubleshooting and analysis.
Use Cases:
- Track accuracy of predictions as actuals come in
- Monitor specific slices of predictions
- Track performance for a subset of features
- Detect when Type I (false positive) and Type II (false negative) error rates surpasses a specified threshold
- Glean insights into when a model needs retraining due to performance degradation
The performance metrics supported vary based on the model type sent in. Visit this page to learn more about model types supported and metrics associated with each.
All monitors, including performance monitors, can be set up directly from the Monitors Page or from any Model Overview page. To learn more visit the Set up Model Monitors page.
Click 'Create' to begin configuring a new performance monitor. Performance can be monitored for any combination of dimensions and prediction slices. In the example below, the monitor for model wine_quality is set to track accuracy when feature alcohol is greater than 12 AND feature ash is greater than 2 in the last rolling 30 day window:
Performance Monitor Configurator
Once you are finished configuration the monitor, click 'Save Changes' and you're all set!
Automatic thresholds are set by Arize when there is sufficient production data to determine a trend. The threshold is determined by looking back at a historical time window for a metric and calculating the variance of data in that time period (more here). Automatic thresholding is enabled by default for new monitors. Users can toggle automatic thresholds on or off from the “Edit monitor” configuration.
With auto thresholds turned off, the user is free to set the threshold to any value. We display the mean and standard deviation values used to calculate the auto threshold, and the user can change the number of standard deviations above or below the mean to calculate a suggested threshold.
Last modified 22d ago