Learn how to setup an import job using Databricks
This feature is currently in beta release, please contact [email protected] for access

Step 1 - Start the Data Upload Wizard

Navigate to the 'Upload Data' page on the left navigation bar in the Arize platform. From there, select the 'Databricks' card or navigate to the Data Warehouse tab to start a new table import job to begin a new table import job.
Storage Selection: Databricks
Select Databricks from Table Options

Step 2 - Input the Project ID, Dataset, and Table / View

Locate the Project ID, Dataset, and Table or View name of the table/view you would like to sync from Databricks.
  • The Databricks profile_path is a unique identifier for a project. See here for steps on how to retrieve this ID using Unity catalog.
  • The dataset and table name correspond to the path where your table is located
Fill in the table information

Step 3 - Grant Access To Your Dataset, Table, or View

Tag your dataset/table/view with the arize-ingestion-key and the provided label value using the steps below. For more details, see docs on Adding labels to resources for Databricks.
In Arize UI: Copy arize-ingestion-key value
Copy your ingestion key and add it as a key value pair to your Databricks dataset
If you have issues granting permissions please reach out to [email protected]

Step 4 - Configure Your Model And Define Your Table’s Schema

Match your model schema to your model type and define your model schema through the form input or a json schema.
Set up model configurations
Map your table using a form
Map your table using a JSON schema
The unique identifier of a specific prediction. Limited to 128 characters.
The timestamp of the prediction in seconds or an RFC3339 timestamp
Optional, defaults to current timestamp at file ingestion time
Timestamp of when a row was added (see example for details)
Required *(only applicable for table upload)
Column name for the prediction value
Required based on model type
Column name for the predicted score
Required based on model type
Column name for the actual or ground truth value
Optional for production records
Column name for the ground truth score
Required based on model type
Column name for ranking groups or lists in ranking models
Required for ranking models
Column name for rank of each element on the its group or list
Required for ranking models
Column name for ranking actual or ground truth value
Required for ranking models
Column name for ranking ground truth score
Required for ranking models
A string prefix to describe a column feature/. Features must be sent in the same file as predictions
Arize automatically infers columns as features. Choose between feature prefixing OR inferred features.
A string prefix to describe a column tag/. Tags must be sent in the same file as predictions and features
A string prefix to describe a column shap/. SHAP must be sent in the same file as predictions or with a matching prediction_id
A column to specify model version. version/ assigns a version to the corresponding data within a column, or configure your version within the UI
Optional, defaults to 'no_version'
Distinguish different batches of data under the same model_id and model_version. Must be specified as a constant during job setup or in the schema
Optional for validation records only
A list of columns to exclude if the features property is not included in the ingestion schema
A list of embedding columns, required vector column, optional raw data column, and optional link to data column. Learn more here
Once finished, Arize will begin querying your table and ingesting your records as model inferences.

Step 5 - Add Model Data To The Table Or View

Arize will run queries to ingest records from your table based on your configured refresh interval.

Step 6 - Check your Table Import Job

Arize will attempt a dry run to validate your job for any access, schema or record-level errors. If the dry run is successful, you may then create the import job.
After creating a job following a successful dry run, you will be taken to the 'Job Status' tab where you can see the status of your import jobs.
You can view the job details and import progress by clicking on the job ID, which uncovers more information about the job.

Step 7 - Troubleshooting An Import Job

An import job may run into a few problems. Use the dry run and job details UI to troubleshoot and quickly resolve data ingestion issues.

Validation Errors

If there is an error validating a file or table against the model schema, Arize will surface an actionable error message. From there, click on the 'Fix Schema' button to adjust your model schema.

Dry Run File/Table Passes But The Job Fails

If your dry run is successful, but your job fails, click on the job ID to view the job details. This uncovers job details such as information about the file path or query id, the last import job, potential errors, and error locations.
Once you've identified the job failure point, append the edited row to the end of your table with an updated change_timestamp value.