Arize AI
Search…
User Guides
Integrations
On-Premise Deployment
CML (DVC)
Example of Arize use in CI / CD Workflow
Overview
This tutorial runs through how to use Arize in a Continuous Integration and Continuous Deployment workflow for models.
This tutorial is based on Continuous Machine Learning Groups work:
CML · Continuous Machine Learning
CML · Continuous Machine Learning
This tutorial will show you how to:
- Integrate Arize into the CI/CD workflow
- Run Arize on Validation data every time a new model version is checked in
- Capture data in the platform to compare different validated models
Workflow Overview
The CI/CD workflow for models with CML involves a training script and a linkage to Github actions.
CI CD CML Example
The following describes the steps for training and validation runs:
- 1.A model directory is setup on Github which contains both model file and train scripts for CML
- 2.Train scripts are built to run a set of inferences across any newly built model
- 3.GitHub actions are setup to run the train script on any model checkin
- 4.On Model Checkin the train script is run
- 1.The train Script logs the validation inferences to Arize
- 2.Checks within the Arize platform can be setup to run on every new validation batch of data. These checks can include comparing against previous model data or fixed levels analysis
- 3.On check failure dashboards can be created for model analysis
- 4.Future: The ability to quickly poll through API the validation checks as part of Github actions for pass / fail
An example Train script for CML with Arize is included here:
The github/workflows directory defines the github actions that are run on model checkin. This is derived from the CML example.
client_python/train.py at main · Arize-ai/client_python
GitHub
The Train scripts typically have 2 parts:
- 1.Train and Score the latest model
1
#################################
2
########## MODELLING ############
3
#################################
4
5
# Fit a model on the train section
6
regr = RandomForestRegressor(max_depth=2, random_state=seed)
7
regr.fit(X_train, y_train)
8
9
# Report training set score
10
train_score = regr.score(X_train, y_train) * 100
11
# Report test set score
12
test_score = regr.score(X_test, y_test) * 100
13
y_pred = regr.predict(X_test)
Copied!
2. Log artifacts and data to CML
1
# Write scores to a file
2
with open("metrics.txt", 'w') as outfile:
3
outfile.write("Training variance explained: %2.1f%%\n" % train_score)
4
outfile.write("Test variance explained: %2.1f%%\n" % test_score)
Copied!
An additional 3rd section is added to send the feature data, inferences (predictions) and ground truth (actuals) to Arize
1
#############################################
2
########## Arize AI Validation Sample ############
3
#############################################
4
5
SPACE_KEY="SPACE_KEY"
6
API_KEY="API_KEY"
7
model_name = "validation-wine-model-cicd"
8
9
10
datetime_rightnow = datetime.datetime.today()
11
model_version_id_now = 'train_validate_' + datetime_rightnow.strftime('%m_%d_%Y__%H_%M_%S')
12
id_df = pd.DataFrame([str(id) + model_version_id_now for id in X_test.index])
13
arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY,uri='https://devr.arize.com/v1')
14
tfuture = arize_client.log(model_id=model_name, model_version=model_version_id_now,
15
features=X_test, prediction_ids=id_df,
16
prediction_labels=pd.DataFrame(y_pred))
17
tfuture = arize_client.log(model_id=model_name, model_version=model_version_id_now,
18
prediction_ids=id_df, actual_labels=pd.DataFrame(y_test))
Copied!
If using version < 4.0.0, replace
space_key=SPACE_KEY with organization_key=SPACE_KEYThe above workflow can be modified for any model CI CID flow where scoring is done from a set of validation inferences.
Last modified 3mo ago
Copy link