Monitors API
Learn how to query, patch, and create monitors using our programmatic API
Monitors keep track of the various facets of your models. They track key metrics over time and let you analyze incidents that may be causing your model to degrade. The Arize platform provides workflows to troubleshoot incidents whenever a monitor is triggered.
However, it's also useful to export existing monitors programmatically and make configuration changes outside of the Arize platform. The Arize GraphQL API is designed to do just that - to allow you to build tooling that lets you manage monitors using your own custom workflows.
Monitors belong to a
Model and thus also belong to a Space. Because of this, you can either query for monitors off of a Model node or a Space node.The model or space id is in the url when you visit the corresponding page. The format looks like this:
spaces/:space_id/models/:model_id.Query by Model
Query by Space
query {
node(id: "model_id") {
... on Model {
monitors(first: 50) {
edges {
node {
id
name
monitorCategory
}
}
}
}
}
}
query {
node(id: "space_id") {
... on Space {
monitors(first: 50) {
edges {
node {
id
name
monitorCategory
}
}
}
}
}
}
The
MonitorsConnection off of Model and Space provide a convenient way for you to pull monitors that match the given criteria. If you have a large number of monitors, you will have to use pagination to pull the complete list. This data can then be structured to your liking and be exported to the platform of your choosing. If you are managing a large number of monitors with Arize, you may want to periodically update certain parts of the configuration of a group of monitors. The Arize API provides
patch mutations (partial modification) for each type of monitor. Here are a few examples:
Patch Drift Monitor
Patch Performance Monitor
Each
patch mutation takes a monitor's node_id an a set input that contains the properties of the monitor. Each mutation returns the monitor that has been modified - there is no need to re-query for the data! View an example Colab below.
Google Colaboratory
Patching Drift Monitors
Notice that the mutations above make use of naming a query and using variables to make the mutations re-usable.
This is a very powerful abstraction and will help you construct maintainable queries and mutations. See GraphQL docs
Deleting monitors using a mutation simply requires you to know the
ID of the monitor you want to delete.mutation {
deleteMonitor(input: {monitorId: "monitor_id"}) {
clientMutationId
}
}
You can mute a monitor by using the generic
patchMonitor or a specific mutation like patchDriftMonitor, patchPerformanceMonitor, or patchDataQualityMonitor.Generic mutation
Specific mutation
mutation {
patchMonitor(
input: {
monitorId: "monitor_id"
set: {
notificationsEnabled: false
}
}) {
clientMutationId
}
}
mutation {
patchMonitor(
input: {
monitorId: "monitor_id"
set: {
notificationsEnabled: false
}
}) {
clientMutationId
}
}
In most cases, the Arize platform will provide you with the best experience for creating monitors as the UI gives you a live preview of your monitor as you fill out its configuration. Additionally, Arize's managed monitors, auto-thresholding capabilities, and model baselines can often provide great monitoring coverage for your models.
However, there may be cases where you would like to create a known set of custom monitors that track a specific cohort (filtered dataset) of features or a set of monitors that track a specific tag. Programmatically creating monitors can provide you the flexibility you need so you can ensure sufficient monitoring coverage.
Let's suppose you have a lending model and you would like to exhaustively monitor the performance of the model for all 50 states. You may want to create a re-usable mutation that looks something like
mutation createPerformanceMonitorForState($state: String!){
createPerformanceMonitor(
input: {
modelId: "model_id",
operator: lessThan,
performanceMetric: accuracy,
dynamicAutoThreshold: {
stdDevMultiplier: 1.2
}
filters: [{
dimensionType: featureLabel,
operator: equals
name: "stateName"
values: [$state]
}],
contacts: [{
notificationChannelType: email,
emailAddress: "[email protected]"
}]
}
) {
monitor {
id
}
}
}
Notice that we have used a variable to pass in a state to the
createPerformanceMonitor mutation. We now can loop over all 50 states, invoke this mutation and get 100% coverage for all 50 states. The end result will be 50 monitors that will alert you when the performance dips below the specified threshold.Two different variants of thresholds are supported: fixed thresholds and dynamic auto thresholds. Fixed thresholds are a great fit when you explicitly know when you would like the monitor to fire. However, if you would like the monitor to automatically calculate a threshold for you, you can supply an auto-threshold configuration. Note that by default, auto thresholds are enabled with a standard deviation multiplier of 2.
Auto Threshold
Manual Threshold
dynamicAutoThreshold: {
stdDevMultiplier: 2
}
threshold: 0.01
This configuration will set your threshold to
mean +/- (stdDevMultiplier * stdDev) of a historical sample. The +/- part of the threshold calculation is determined by the operator.Having trouble? Reach out to us via email [email protected] or Slack us in the #arize-support channel for more support.
The
filters list lets you filter down the metric you are monitoring down to a cohort of inferences. It lets you filter down the data you are monitoring by features, tags, and actuals. Here are some examples:Features and Tags
Predictions
Actuals
GraphQL Type
filters: [
{
dimensionType: featureLabel,
name: "stateName"
operator: equals
values: ["california"]
},
{
dimensionType: tagLabel,
name: "business_impact_score"
operator: greaterThan
values: ["2"]
}
]
filters: [
{
dimensionType: predictionClass,
operator: equals
values: ["fraud"]
},
{
dimensionType: predictionScore,
operator: lessThan
values: ["0.7"]
}
]
filters: [
{
dimensionType: actualClass,
operator: equals
values: ["fraud"]
},
{
dimensionType: actualScore,
operator: lessThan
values: ["0.7"]
}
]
"""Filters for a particular model's inference data"""
input DimensionFilterInput {
"""The part of the model to filter on."""
dimensionType: DimensionFilterType!
"""The operator of the filter."""
operator: ComparisonOperator!
"""Required when specifying a tag or a feature label filter."""
name: String
"""The values to match on."""
values: [String!]!
}
Note that numeric filter values are passed in as strings but they must be parsable as numbers.
Drift is monitored by comparing your model's current distribution to a baseline distribution. For more information on baselines and how to choose a baseline, refer to our documentation here. There are a few ways to set a baseline using our API:
Model Baseline
Custom Moving Baseline
Custom Fixed Baseline
A model baseline refers to the baseline used for all monitors in a model. Your model is preset with a baseline defined by a moving time range of your model's production data for 17 days.
mutation {
createDriftMonitor(input: {
modelId: "model_id"
dimensionCategory: predictionClass
name: "Drift Monitor using Moving Custom Baseline"
operator: greaterThan
driftMetric: psi
baseline: {
useModelPrimaryBaseline: true,
}
}) {
monitor {
uri
}
}
}
Custom baselines can be set for individual monitors separate of the overall model baseline. In this case, the baseline utilizes a moving time window.
mutation {
createDriftMonitor(input: {
modelId: "model_id"
dimensionCategory: predictionClass
name: "Drift Monitor using Moving Custom Baseline"
operator: greaterThan
driftMetric: psi
baseline: {
useModelPrimaryBaseline: false,
movingWindowSeconds: 604800, // 1 week
movingWindowDelaySeconds: 864000, // 10 days
}
}) {
monitor {
uri
}
}
}
Custom baselines can be set for individual monitors separate of the overall model baseline. In this case, the baseline utilizes a fixed time window.
mutation {
createDriftMonitor(input: {
modelId: "model_id"
dimensionCategory: predictionClass
name: "Drift Monitor using Fixed Custom Baseline"
operator: greaterThan
driftMetric: psi
baseline: {
useModelPrimaryBaseline: false,
fixedStartDate: "2020-12-23T04:59:59.999Z",
fixedEndDate: "2021-12-23T04:59:59.999Z",
}
}) {
monitor {
uri
}
}
}
Google Colaboratory
Creating Performance Monitors
Integrations belong to a
Model and thus also belong to a Space. Because of this, you can either query for monitors off of a Model node or a Space node.The model or space id is in the url when you visit the corresponding page. The format looks like this:
spaces/:space_id/models/:model_id.Query by Model
Query by Space
query {
node(id: "model_id") {
... on Model {
monitors(first: 50) {
edges {
node {
id
name
monitorCategory
}
}
}
}
}
}
query {
node(id: "space_id") {
... on Space {
monitors(first: 50) {
edges {
node {
id
name
monitorCategory
}
}
}
}
}
}
Monitors surface potential issues with performance and data, sending real-time alerts so you can take immediate action. Learn how to programmatically update email alerts and integrate with other altering systems.
You can update either an email or paging service your monitor alerts by using the generic
patchMonitor or a specific mutation such as: patchDriftMonitor, patchPerformanceMonitor, or patchDataQualityMonitor. Generic mutation
mutation {
patchMonitor(
input: $input) {
clientMutationId
}
}
variables
{
"monitorId": ":monitor_id",
"set": {
"contacts": [
{
"emailAddress": "[email protected]",
"notificationChannelType": "email"
},
{
"integrationKeyId": ":integration_id",
"notificationChannelType": "integration"
}
]
}
}
Natively integrate with an alerting service to identify and resolve issues faster. Learn more about our alerting integrations here.
Integrations belong to an
AccountOrganization, thus belonging to an Account. Because of this, you must query for integrations off of an AccountOrganization node.You can find the account organization in the url organizations/:account_organization_id/Query
Create
Patch
Delete
Test Integration
query GetIntegrationsForOrganization {
node(id: "account_organization_id") {
... on AccountOrganization {
integrations {
id
providerName
name
}
}
}
}
mutation CreateIntegrationKey($input: CreateIntegrationKeyInput!) {
createIntegrationKey(input: $input) {
integrationKey {
providerName
name
id
}
}
}
variables
{
"input": {
"providerName": "provider_name",
"accountOrganizationId": "account_organization_id",
"serviceName": "your_custom_service_name",
"apiKey": "api_key_from_provider"
}
}
mutation PatchIntegrationKey($input: PatchIntegrationKeyInput!) {
patchIntegrationKey(input: $input) {
integrationKey {
providerName
name
id
}
}
}
variables
{
"input": {
"integrationKeyId": "integration_key_id",
"set": {
"serviceName": "new custom service name",
"apiKey": "new_api_key_from_provider"
}
}
}
mutation DeleteIntegrationKey {
deleteIntegrationKey(input: {integrationKeyId: "integration_key_to_delete"}) {
integrationKey {
providerName
name
id
}
}
}
mutation {
testIntegrationKey(
input: {
integrationKeyId: "integration_key_id"
}
) {
statusCode
}
}
Last modified 12d ago