Monitor Drift, Performance, and Data Quality issues of your Model
Monitors automatically detect drift, data quality issues, or anomalous performance degradations with highly configurable monitors based on both common KPIs and custom metrics. Model monitoring empowers ML Engineers to quickly detect issues and pinpoint where to begin further analysis for quick resolution.
Monitors surface potential issues with performance and data, sending real-time alerts so you can take immediate action. You can set up any of the following types of monitors in the Arize platform:
Data and Model Drift: Monitor for drift to identify if your models have grown stale, you have data quality issues, or if there are adversarial inputs in your model. Detecting drift with ML monitoring will help protect your models from performance degradation and allow you to better understand how to begin resolution.
Data Quality: Model performance is highly dependent on the quality of the data sources powering your model’s features; use ML monitoring to identify cardinality shifts, data type mismatch, missing data, and more to improve data quality.
Performance: Model performance metrics measure how well your model performs in production. Monitor model performance with daily or hourly checks on metrics such as Accuracy, Recall, Precision, F1, MAE, or MAPE.
Monitors are continuously evaluated according to each monitor's configurations -- an evaluation will result in one of the following statuses for a monitor, ordered by severity:
Triggered - The monitor's alert conditions have been violated.
NoData - There is insufficient model data to check if there is a violation of the monitor's conditions.
Healthy - The monitor's alert conditions have not been violated.
When viewing the health of your model for a monitor type (drift, performance, or data quality), the status shown will be based on the highest severity of any of the model's monitors for that type. For example, if you are monitoring your model with several performance monitors, such as Accuracy and F1-score, and at least one of them has a "Triggered" status, the status shown for the model's performance will be "Triggered".
There will be an additional possible status when viewing your model's health:
Not Monitored - Model is not being monitored by a particular monitor type.
A monitor's threshold is the value that is compared against your model's current calculated metric value.Thresholds are used to trigger an alert when the current value of a metric is either above or below a model's threshold value. Arize automatically calculates and sets thresholds upon creation of managed drift or performance monitors.
Setting Up Monitors
There are two different categories of monitors on the Arize platform: managed monitors and custom monitors. Managed monitors are the default set of monitors associated with your model type. We recommend using managed monitors if you're new to the platform because they provide a holistic view of your data quality, drift, and model performance. From there, you can customize our default (managed) monitors, or create your own monitors from scratch.
Setting Up Monitors: Automatic (Managed) Monitors
You can set up managed monitors via the 'Model Overview' page or 'Monitors' tab. From the 'Model Overview' page, scroll down to the 'Monitors' card and click on the 'Set Up Monitors' button. A pop-up will appear with the option to choose an evaluation metric and positive class for performance monitors, the option to disable a monitor type, and a drop-down menu to configure your alerts.
You can also set up managed monitors from the 'Monitors' tab. Click on the 'Set Up Monitors' button to follow the same flow as above.
Setting up Monitors: Custom Monitors
You can customize monitors from scratch or once your monitors are all set from the managed monitors flow. Custom monitors allow you to change your threshold, edit your evaluation metric, filter, and ore.
Pro tip: If you're unsure what monitors are relevant for your model, use managed monitors as a template to decide how to fine-tune your monitors.
Custom Monitor from a Managed Managed Monitor
The easiest way to customize a monitor is from an existing monitor. Click into the monitor you want to customize from the 'Monitors' tab, and click 'Edit Monitor' on the top right which will prompt you to duplicate your monitor for customization.
Configure elements such as:
Metric: edit based on a wide range of metrics such as F_1, AUC, RMSE, and more
Filters: filter your monitor on prediction score, feature, actual class, etc.
Evaluation window: change the time window from 1 hour - 30 days
Threshold value: automatic or custom, edit the multiplier within the calculated value
Who to alert: change your integration or email alerts
New Custom Monitor
To create a net new custom monitor, navigate to the 'Monitors' tab and select 'New Monitor' on the top right to customize your model details.
To close the loop, set up an alerting integration with systems such as Pagerduty, Slack, OpsGenie, and Amazon Eventbridge integrations to notify you when a monitor has been triggered. Learn more below: