Google BigQuery FAQ

Here you will find a list of common questions regarding data warehouse jobs

What types of BigQuery tables are supported?

Arize can ingest data from BigQuery views or native tables. Use cloud storage to ingest external tables.

How do I use change_timestamp?

The change_timestamp field is used for Arize to track which new rows need to be imported as we constantly sync new data. This column type may be the following types: TIMESTAMP, DATE, or DATETIME. Unless your table is small, it is important for performance and cost to ensure this column is also partitioned, so we don't need to re-scan data. For best performance, values in the change_timestamp field should also be unique.

prediction_id	change_timestamp	predicted_label	feature_a
1	2023-01-27 17:57:59.932798 UTC	Spam	13
2	2023-02-15 12:01:00.134899 UTC	Not Spam	15
3	2023-02-15 12:15:00.134899 UTC	Spam	3

When adding new data, Arize will only ingest data “newer” (based on change_timestamp column) than what's currently ingested.

Can I update or delete rows?

Deleted or updated data in your data warehouse will not be reflected by Arize. Only new records added to your data warehouse table will be ingested.

How do I update/fix a row that failed to ingest?

If your query failed due to an issue with a specific row or set of rows, fix the error and append the data to your table again using an updated change_timestamp.

How do I debug my failed BigQuery table query?

Any queries that fail to run for a given table import job will be displayed within the job's details.

Sample Query Failure in Job Details

The QueryID that is displayed corresponds to a BigQuery job id. The BigQuery job id can be used to retrieve the execution of a given query and lookup the specific query to rerun.

There are two ways to look up additional information about a BigQuery job.

1. BigQuery CLI

bq show -j --format=prettyjson <project id>:<region>.<job id>

The results of this command will display the query that ran for the given job, query parameters that were applied, error statuses, and information about its execution cost.

1. BigQuery UI

Job statuses can be retrieved for a given BigQuery project.

After searching for and selecting the given job, BigQuery will provide information about the query that was run and when it was run. However, it does not fill in the values of any query parameters that are used in the query. This information is provided by the Job Details page in Arize, and can be used to recreate the query.