Drift monitors allows for tracking of feature drift and model prediction drift from a baseline sample.
The core metric used for drift monitors is a modified version of Population Stability Index (PSI). The algorithm and modifications are covered in the whitepaper linked at the bottom of this page. The key modifications allow for graceful handling of out of distribution events for both categorical and numeric distributions.
The different areas of monitoring drift in the platform fall into three categories:
- Feature Drift
- Prediction Drift
- Actuals Drift (Concept Drift Tracking)
Feature and Data Tracking
For monitoring unstructured data, users can create embedding drift monitors. Embedding drift is tracked by calculating the Euclidean Distance between embedding vectors. To create an embedding drift monitor simply follow the flow above and select an embedding feature in the "Drift of" section of the monitor form. After creating an embedding drift monitor, click "Troubleshoot Drift" to navigate to the embedding details page for further troubleshooting (more here).
Automatic thresholds are set by Arize when there is sufficient production data to determine a trend. The threshold is determined by looking back at a historical time window for a metric and calculating the variance of data in that time period (more here). Automatic thresholding is enabled by default for new monitors. Users can toggle automatic thresholds on or off from the “Edit monitor” configuration.
With auto thresholds turned off, the user is free to set the threshold to any value. We display the mean and standard deviation values used to calculate the auto threshold, and the user can change the number of standard deviations above or below the mean to calculate a suggested threshold.
The above view shows both the historical tracking of Population stability index (PSI) monitor and the current distribution view that generated the PSI value.
In the Metric History view, the data over time is historical based on the monitor data. The historical time series view allows teams to visualize the change in PSI over the period of time the monitor was enabled:
The distribution view allows teams to understand the underlying distribution that created the PSI measure. The distribution view shows both a reference (baseline) comparison to a production distribution.
Statistical Distances for ML Update.pdf