Delayed Actuals
Connect model predictions to delayed ground truth data
What Are Delayed (Latent) Actuals
Depending on your model use case, you may experience a delayed feedback loop when collecting ground truth data. We call this data delayed actuals.
If your model receives delayed actuals, Arize can automatically connect actuals to predictions sent earlier via the same prediction ID.
Sending Delayed Actuals
Utilize the Arize joiner to easily match delayed actuals with predictions in the Arize platform. To do this, simply upload your actuals data using the same prediction_id
as its corresponding prediction.
Joiner Cadence & Lookback
The Arize joiner automatically triggers daily at 05:00 UTC to map delayed actuals with their corresponding prediction values up to 14 days from when the prediction was received. This is supported for all data upload methods.
Joins are conducted on actuals sent within the join window for the day prior, which is from 00:00 UTC to 23:59 UTC.
The Arize support team can extend your 14-day connection window and increase your joiner cadence upon request. Reach out to support@arize.com for help.
Joiner Requirements
Upload delayed actuals for ranking models with file/table upload via GraphQL or SDK. Native UI upload support coming soon. Reach out to support@arize.com for help and questions.
Example Joins By Upload Method
To send delayed actuals via GCS, AWS S3, Azure Blob Storage, Google BigQuery, and Snowflake, configure separate data ingestion jobs for predictions and actuals. We recommend naming job prefixes to indicate which job contains predictions or actuals.
Make sure that your prediction ID, model name, and space match with your corresponding predictions when defining the schema for these two data ingestion jobs. Once you configure both jobs, Arize will automatically recognize and sync new prediction and actual data. To validate new data in Arize, visualize the data in the 'Dataset' tab.
Tags with Delayed Actuals
Tags can be updated via delayed actuals. If tags are sent with actuals, the tag will be joined based on prediction_id
. However, if the actual was sent prior, then resent with an updated tag value, the tag value will not be updated.
Tags will remain the same if they are sent with predictions, not actuals.
For example, if a user sends Arize a prediction with tags:
And actual with tags:
The resulting tags available will be:
Measure Model Performance
Arize only calculates performance metrics on predictions that have actuals, so once your join is represented in Arize, you can utilize performance metrics and the 'Performance Tracing' tab for those predictions.
If actuals have not been received yet (delayed actuals), use drift as a proxy metric for model performance to measure and monitor model health.
Last updated