How To Send Delayed Actuals
Connect model predictions to delayed ground truth data
Depending on your model use case, you may experience a delayed feedback loop when collecting ground truth data. We call this data delayed actuals.
If your model receives delayed actuals, Arize can automatically connect actuals to predictions sent earlier via the same prediction ID.
Utilize the Arize joiner to easily match delayed actuals with predictions in the Arize platform. To do this, simply upload your actuals data using the same
prediction_id as its corresponding prediction.
Example join with a 7-day lookback window
The Arize joiner automatically triggers daily at 05:00 UTC to map delayed actuals with their corresponding prediction values up to 14 days from when the prediction was received. This is supported for all data upload methods.
Joins are conducted on actuals sent within the join window for the day prior, which is from 00:00 UTC to 23:59 UTC.
The Arize support team can extend your 14-day connection window and increase your joiner cadence upon request. Reach out to [email protected] for help.
Field | Description |
|---|---|
prediction_id | (required) A prediction's unique identifier. The actual's prediction_id must match its corresponding prediction to join the data |
actual_score / actual_label | (required) The ground truth values of your model. The use of score and label varies based on model type |
model_id | (required) When sending delayed actuals, specify the model_id in your schema to match your actuals to the correct model |
Field | Description |
|---|---|
prediction_group_id | (required) A ranking prediction group's unique identifier. The actual's prediction_group_id must match its corresponding prediction_group_id to join the data, without any nulls |
rank | (required) A unique value within each ranking prediction group limited from 1-100 |
relevance_score/relevance_label | (required) The ground truth values specific to ranking models |
Cloud Storage/ Data Lake
Python Pandas
Python Single Record
There are two ways to send delayed actuals via GCS, AWS S3, Azure Blob Storage, and Google BigQuery - depending on your use case.
Send delayed actuals using the same job as your predictions. To do this, add your actuals under a subfolder using the same prefix. In this case
gs://bucket1/click-thru-rate/production/This method enables you to automatically sync your new data as Arize already recognizes and checks the existing job for new data. Once Arize recognizes delayed actuals data, you can visualize feedback in the platform in the model's 'Dataset' tab.
gs://bucket1/click-thru-rate/production/
├── prediction-folder/
│ ├── 12-1-2022.parquet #this file can contain multiple versions
│ ├── 12-2-2022.parquet
│ ├── 12-3-2022.parquet
├── actuals-folder/
│ ├── 12-1-2022.parquet
│ ├── 12-2-2022.parquet
│ └── 12-3-2022.parquet
Option 2: Setup A New Job To Upload Delayed Actuals
If you can't write to the same directory multiple times, have delayed tags, or separate your environments, create a new import job to upload delayed actuals. We recommend naming job prefixes that indicate which job contains predictions or actuals.
In this case, make sure your prediction id, model name, and space match with your corresponding predictions.
gs://bucket1/click-thru-rate/prediction/
├── 11-19-2022.parquet
├── 11-20-2022.parquet
├── 11-21-2022.parquet
gs://bucket1/click-thru-rate/actuals/
├── 12-1-2022.parquet # same prediction id column, model, and space as the corresponding prediction
├── 12-2-2022.parquet
└── 12-3-2022.parquet
To log delayed actuals using the Python SDK, simply match the actuals
prediction_id_column_name with its corresponding prediction. From there, Arize will automatically identify the join and match the data together. #log predictions
schema = Schema(
prediction_id_column_name="prediction_id",
prediction_label_column_name="prediction_label",
...
)
# then log actuals
schema = Schema(
prediction_id_column_name="prediction_id", #needs to be the same as above
actual_label_column_name="actual_label",
...
)
To log delayed actuals using the Python Single Record SDK, simply match the actual
prediction_id with its corresponding prediction. From there, Arize will automatically identify the join and match the data together. #log the features & prediction
response = arize.log(
prediction_id='plED4eERDCasd9797ca34',
model_id='sample-model-1',
model_type=ModelTypes.SCORE_CATEGORICAL,
environment=Environments.PRODUCTION,
model_version='v1',
prediction_timestamp=1618590882,
features=features,
prediction_label=('Fraud',.4),
tags=tags
)
#log the actual
actual_response = arize.log(
prediction_id='plED4eERDCasd9797ca34',
model_id='sample-model-1',
model_type=ModelTypes.SCORE_CATEGORICAL,
environment=Environments.PRODUCTION,
actual_label=('Fraud',1),
tags=tags)
Tags can be updated via delayed actuals. If tags are sent with actuals, the tag will be joined based on
prediciton_id. However, if the actual was sent prior, then resent with an updated tag value, the tag value will not be updated. Tags will remain the same if they are sent with predictions, not actuals.
For example, if a user sends Arize a prediction with tags:
"location": "New York"
"month": "January"
And actual with tags:
"location": "Chicago"
"fruit": "apple"
The resulting tags available will be:
"location": "New York"
"month": "January"
"fruit": "apple"
Arize only calculates performance metrics on predictions that have actuals, so once your join is represented in Arize, you can utilize performance metrics and the 'Performance Tracing' tab for those predictions.
If actuals have not been received yet (delayed actuals), use drift as a proxy metric for model performance to measure and monitor model health.