Monitor Drift, Performance, and Data Quality issues of your Model
Monitors automatically detect drift, data quality issues, or anomalous performance degradations with highly configurable monitors based on both common KPIs and custom metrics. Model monitoring empowers ML Engineers to quickly detect issues and pinpoint where to begin further analysis for quick resolution.
Monitors surface potential issues with performance and data, sending real-time alerts so you can take immediate action. You can set up any of the following types of monitors in the Arize platform:
Data and Model Drift: Monitor for drift to identify if your models have grown stale, you have data quality issues, or if there are adversarial inputs in your model. Detecting drift with ML monitoring will help protect your models from performance degradation and allow you to better understand how to begin resolution.
Data Quality: Model performance is highly dependent on the quality of the data sources powering your model’s features; use ML monitoring to identify cardinality shifts, data type mismatch, missing data, and more to improve data quality.
Performance: Model performance metrics measure how well your model performs in production. Monitor model performance with daily or hourly checks on metrics such as Accuracy, Recall, Precision, F1, MAE, or MAPE.
Monitors are continuously evaluated according to each monitor's configurations -- an evaluation will result in one of the following statuses for a monitor, ordered by severity:
Triggered - The monitor's alert conditions have been violated.
NoData - There is insufficient model data to check if there is a violation of the monitor's conditions.
Healthy - The monitor's alert conditions have not been violated.
When viewing the health of your model for a monitor type (drift, performance, or data quality), the status shown will be based on the highest severity of any of the model's monitors for that type. For example, if you are monitoring your model with several performance monitors, such as Accuracy and F1-score, and at least one of them has a "Triggered" status, the status shown for the model's performance will be "Triggered".
There will be an additional possible status when viewing your model's health:
Not Monitored - Model is not being monitored by a particular monitor type.
A monitor's threshold is the value that is compared against your model's current calculated metric value.Thresholds are used to trigger an alert when the current value of a metric is either above or below a model's threshold value. Arize automatically calculates and sets thresholds upon creation of managed drift or performance monitors.
Setting Up Monitors
We recommend starting with the default set of monitors associated with your model type. There are multiple ways to set up a monitor.
You can set up your model to be automatically monitored when you set up your model by going to the blue banner and clicking "Set up Model".
You can also set up custom monitors by navigating to the Monitors tab on the left hand side, select "New Monitor", and then choose which model to apply it to.
You can also navigate to the Models page and select your desired model, to get to the Model Overview page.
From there you can select the Monitors tab at the top, or on the Monitor module on the right side, and click "New Monitor".
When setting up or editing a monitor, you can configure elements such as Metric, Dimension, Aggregation, Filters (e.g. prediction score, feature, actual class, etc.), evaluation window, threshold value (automatic or custom), when to trigger an alert, and who to email when it is triggered.
Additionally, while editing, the monitor's metric history is simulated to best reflect your configuration changes.
Monitors can trigger email alerts. The email alerts can be used with a Pagerduty, Slack, OpsGenie, Amazon Eventbridge integrations.