Programmatically Create Monitors

Learn how to query, patch, and create monitors using our programmatic API

Monitors keep track of the various facets of your models. They track key metrics over time and let you analyze incidents that may be causing your model to degrade. The Arize platform provides workflows to troubleshoot incidents whenever a monitor is triggered.

However, it's also useful to export existing monitors programmatically and make configuration changes outside of the Arize platform. The Arize GraphQL API is designed to do just that - to allow you to build tooling that lets you manage monitors using your own custom workflows.

For a brief overview of GraphQL itself, please consult our introduction.

Querying for Monitors

Monitors belong to a Model and thus also belong to a Space. Because of this, you can either query for monitors off of a Model node or a Space node.

The model or space id is in the url when you visit the corresponding page. The format looks like this: spaces/:space_id/models/:model_id.

You can also query for a list of models given the space id. More details here.

query {
  node(id: "model_id") {
    ... on Model {
      monitors(first: 50) {
        edges {
          node {
            id
            name
            monitorCategory
          }
        }
      }
    }
  }
}

query {
  node(id: "space_id") {
    ... on Space {
      monitors(first: 50) {
        edges {
          node {
            id
            name
            monitorCategory
          }
        }
      }
    }
  }
}

The MonitorsConnection off of Model and Space provide a convenient way for you to pull monitors that match the given criteria. If you have a large number of monitors, you will have to use pagination to pull the complete list. This data can then be structured to your liking and be exported to the platform of your choosing.

For a full working example, consult one of our example use case Colabs.

Updating / Patching Monitors

If you are managing a large number of monitors with Arize, you may want to periodically update certain parts of the configuration of a group of monitors. The Arize API provides patch mutations (partial modification) for each type of monitor. Here are a few examples:

mutation updateDriftMonitorThreshold($monitorId: ID!, $threshold: Float!) {
  patchDriftMonitor(
    input: { 
      monitorId: $monitorId, 
      set: { 
        threshold: $threshold, 
        dynamicAutoThresholdEnabled: false 
      }
    }
  ) {
    monitor { id, name, threshold }
  }
}

variables

{
  "monitorId": "monitor_id",
  "threshold": 0.25
}

mutation updatePerfMonitorThreshold($monitorId: ID!, $threshold: Float!) {
  patchPerformanceMonitor(
    input: { 
      monitorId: $monitorId, 
      set: { 
        threshold: $threshold, 
        dynamicAutoThresholdEnabled: false 
      }
    }
  ) {
    monitor { id, name, threshold }
  }
}

variables

{
  "monitorId": "monitor_id"
  "threshold": 0.97
}

Each patch mutation takes a monitor's node_id an a set input that contains the properties of the monitor. Each mutation returns the monitor that has been modified - there is no need to re-query for the data! View an example Colab below.

Notice that the mutations above make use of naming a query and using variables to make the mutations re-usable.

This is a very powerful abstraction and will help you construct maintainable queries and mutations. See GraphQL docs

Deleting Monitors

Deleting monitors using a mutation simply requires you to know the ID of the monitor you want to delete.

mutation {
  deleteMonitor(input: {monitorId: "monitor_id"}) {
    clientMutationId
  }
}

Muting Monitors

You can mute a monitor by using the generic patchMonitor or a specific mutation like patchDriftMonitor, patchPerformanceMonitor, or patchDataQualityMonitor.

mutation {
  patchMonitor(
    input: {
      monitorId: "monitor_id"
      set: {
        notificationsEnabled: false
      }
    }) {
    clientMutationId
  }
}

mutation {
  patchMonitor(
    input: {
      monitorId: "monitor_id"
      set: {
        notificationsEnabled: false
      }
    }) {
    clientMutationId
  }
}

Creating Monitors

In most cases, the Arize platform will provide you with the best experience for creating monitors as the UI gives you a live preview of your monitor as you fill out its configuration. Additionally, Arize's managed monitors, auto-thresholding capabilities, and model baselines can often provide great monitoring coverage for your models. However, there may be cases where you would like to create a known set of custom monitors that track a specific cohort (filtered dataset) of features or a set of monitors that track a specific tag. Programmatically creating monitors can provide you the flexibility you need so you can ensure sufficient monitoring coverage. Let's suppose you have a lending model and you would like to exhaustively monitor the performance of the model for all 50 states. You may want to create a re-usable mutation that looks something like

mutation createPerformanceMonitorForState($state: String!){
  createPerformanceMonitor(
    input: {
      modelId: "model_id", 
      operator: lessThan, 
      performanceMetric: accuracy, 
      dynamicAutoThreshold: {
        stdDevMultiplier: 1.2
      }
      filters: [{
        dimensionType: featureLabel,
        operator: equals
        name: "stateName"
        values: [$state]
      }],
      contacts: [{
        notificationChannelType: email,
        emailAddress: "you@me.com"
      }]
    }
  ) {
    monitor {
      id
    }
  }
}

Notice that we have used a variable to pass in a state to the createPerformanceMonitor mutation. We now can loop over all 50 states, invoke this mutation and get 100% coverage for all 50 states. The end result will be 50 monitors that will alert you when the performance dips below the specified threshold.

Thresholding

Two different variants of thresholds are supported: fixed thresholds and dynamic auto thresholds. Fixed thresholds are a great fit when you explicitly know when you would like the monitor to fire. However, if you would like the monitor to automatically calculate a threshold for you, you can supply an auto-threshold configuration. Note that by default, auto thresholds are enabled with a standard deviation multiplier of 2.

dynamicAutoThreshold: {
   stdDevMultiplier: 2
}

threshold: 0.01

This configuration will set your threshold to mean +/- (stdDevMultiplier * stdDev) of a historical sample. The +/- part of the threshold calculation is determined by the operator.

Having trouble? Reach out to us via email support@arize.com or Slack us in the #arize-support channel for more support.

Monitor Filters

The filters list lets you filter down the metric you are monitoring down to a cohort of inferences. It lets you filter down the data you are monitoring by features, tags, and actuals. Here are some examples:

filters: [
    {
        dimensionType: featureLabel,
        name: "stateName"
        operator: equals
        values: ["california"]
    }, 
    {
        dimensionType: tagLabel,
        name: "business_impact_score"
        operator: greaterThan
        values: ["2"]
    }
]

filters: [
    {
        dimensionType: predictionClass,
        operator: equals
        values: ["fraud"]
    }, 
    {
        dimensionType: predictionScore,
        operator: lessThan
        values: ["0.7"]
    }
]

filters: [
    {
        dimensionType: actualClass,
        operator: equals
        values: ["fraud"]
    }, 
    {
        dimensionType: actualScore,
        operator: lessThan
        values: ["0.7"]
    }
]

"""Filters for a particular model's inference data"""
input DimensionFilterInput {
  """The part of the model to filter on."""
  dimensionType: DimensionFilterType!

  """The operator of the filter."""
  operator: ComparisonOperator!

  """Required when specifying a tag or a feature label filter."""
  name: String

  """The values to match on."""
  values: [String!]!
}

Note that numeric filter values are passed in as strings but they must be parsable as numbers.

Drift Monitor Baselines

Drift is monitored by comparing your model's current distribution to a baseline distribution. For more information on baselines and how to choose a baseline, refer to our documentation here. There are a few ways to set a baseline using our API:

A model baseline refers to the baseline used for all monitors in a model. Your model is preset with a baseline defined by a moving time range of your model's production data for 17 days.

mutation {
  createDriftMonitor(input: {
    modelId: "model_id"
    dimensionCategory: predictionClass
    name: "Drift Monitor using Moving Custom Baseline"
    operator: greaterThan
    driftMetric: psi
    baseline: {
        useModelPrimaryBaseline: true,
    }
  }) {
    monitor {
      uri
    }
  }
}

Custom baselines can be set for individual monitors separate of the overall model baseline. In this case, the baseline utilizes a moving time window.

mutation {
  createDriftMonitor(input: {
    modelId: "model_id"
    dimensionCategory: predictionClass
    name: "Drift Monitor using Moving Custom Baseline"
    operator: greaterThan
    driftMetric: psi
    baseline: {
        useModelPrimaryBaseline: false,
        movingWindowSeconds: 604800, // 1 week
        movingWindowDelaySeconds: 864000, // 10 days
    }
  }) {
    monitor {
      uri
    }
  }
}

Custom baselines can be set for individual monitors separate of the overall model baseline. In this case, the baseline utilizes a fixed time window.

mutation {
  createDriftMonitor(input: {
    modelId: "model_id"
    dimensionCategory: predictionClass
    name: "Drift Monitor using Fixed Custom Baseline"
    operator: greaterThan
    driftMetric: psi
    baseline: {
        useModelPrimaryBaseline: false,
        fixedStartDate: "2020-12-23T04:59:59.999Z",
        fixedEndDate: "2021-12-23T04:59:59.999Z",
    }
  }) {
    monitor {
      uri
    }
  }
}

Querying for Integrations

Integrations belong to a Model and thus also belong to a Space. Because of this, you can either query for monitors off of a Model node or a Space node.

The model or space id is in the url when you visit the corresponding page. The format looks like this: spaces/:space_id/models/:model_id.

You can also query for a list of models given the space id. More details here.

query {
  node(id: "model_id") {
    ... on Model {
      monitors(first: 50) {
        edges {
          node {
            id
            name
            monitorCategory
          }
        }
      }
    }
  }
}

query {
  node(id: "space_id") {
    ... on Space {
      monitors(first: 50) {
        edges {
          node {
            id
            name
            monitorCategory
          }
        }
      }
    }
  }
}

Monitoring Alerts

Monitors surface potential issues with performance and data, sending real-time alerts so you can take immediate action. Learn how to programmatically update email alerts and integrate with other altering systems.

Setting Monitor Alerting Contacts

You can update either an email or paging service your monitor alerts by using the generic patchMonitor or a specific mutation such as: patchDriftMonitor, patchPerformanceMonitor, or patchDataQualityMonitor.

To view and set up external integrations see below.

mutation {
  patchMonitor(
    input: $input) {
    clientMutationId
  }
}

variables

{
  "monitorId": ":monitor_id",
  "set": {
    "contacts": [
      {
        "emailAddress": "example-email-contact@test.com",
        "notificationChannelType": "email"
      },
      {
        "integrationKeyId": ":integration_id",
        "notificationChannelType": "integration"
      }
    ]
  }
}

Alerting Integrations

Natively integrate with an alerting service to identify and resolve issues faster. Learn more about our alerting integrations here.

Querying Current Integrations

Integrations belong to an AccountOrganization, thus belonging to an Account. Because of this, you must query for integrations off of an AccountOrganization node.

You can find the account organization in the url organizations/:account_organization_id/

query GetIntegrationsForOrganization {
  node(id: "account_organization_id") {
    ... on AccountOrganization {
      integrations {
        id
        providerName
        name
      }
    }
  }
}

mutation CreateIntegrationKey($input: CreateIntegrationKeyInput!) {
  createIntegrationKey(input: $input) {
    integrationKey {
      providerName
      name
      id
    }
  }
}

variables

{
  "input": {
    "providerName": "provider_name",
    "accountOrganizationId": "account_organization_id",
    "serviceName": "your_custom_service_name",
    "apiKey": "api_key_from_provider"
  }
}

mutation PatchIntegrationKey($input: PatchIntegrationKeyInput!) {
  patchIntegrationKey(input: $input) {
    integrationKey {
      providerName
      name
      id
    }
  }
}

variables

{
  "input": {
    "integrationKeyId": "integration_key_id",
    "set": {
      "serviceName": "new custom service name",
      "apiKey": "new_api_key_from_provider"
    }
  }
}

mutation DeleteIntegrationKey {
  deleteIntegrationKey(input: {integrationKeyId: "integration_key_to_delete"}) {
    integrationKey {
      providerName
      name
      id
    }
  }
}

mutation {
  testIntegrationKey(
    input: {
      integrationKeyId: "integration_key_id"
    }
  ) {
    statusCode
  } 
}

Examples

API Use Cases

Example Colabs

Patching Drift Monitors

Colab

Creating Performance Monitors

Colab

Creating Data Quality Monitors

Colab

Creating Drift Monitors

Colab

PreviousManual Setup NextBest Practices for Monitors

Last updated 9 months ago