Monitors API

Learn how to query, patch, and create monitors using our programmatic API

Monitors keep track of the various facets of your models. They track key metrics over time and let you analyze incidents that may be causing your model to degrade. The Arize platform provides workflows to troubleshoot incidents whenever a monitor is triggered.

However, it's also useful to export existing monitors programmatically and make configuration changes outside of the Arize platform. The Arize GraphQL API is designed to do just that - to allow you to build tooling that lets you manage monitors using your own custom workflows.

For a brief overview of GraphQL itself, please consult our introduction.

Querying for Monitors

Monitors belong to a Model and thus also belong to a Space. Because of this, you can either query for monitors off of a Model node or a Space node.

The model or space id is in the url when you visit the corresponding page. The format looks like this: spaces/:space_id/models/:model_id.

You can also query for a list of models given the space id. More details here.

Query by Model

query {
  node(id: "model_id") {
    ... on Model {
      monitors(first: 50) {
        edges {
          node {
            id
            name
            monitorCategory
          }
        }
      }
    }
  }
}

Query by Space

query {
  node(id: "space_id") {
    ... on Space {
      monitors(first: 50) {
        edges {
          node {
            id
            name
            monitorCategory
          }
        }
      }
    }
  }
}

The MonitorsConnection off of Model and Space provide a convenient way for you to pull monitors that match the given criteria. If you have a large number of monitors, you will have to use pagination to pull the complete list. This data can then be structured to your liking and be exported to the platform of your choosing.

For a full working example, consult one of our example use case Colabs.

Updating / Patching Monitors

If you are managing a large number of monitors with Arize, you may want to periodically update certain parts of the configuration of a group of monitors. The Arize API provides patch mutations (partial modification) for each type of monitor. Here are a few examples:

Patch Drift Monitor

mutation updateDriftMonitorThreshold($monitorId: ID!, $threshold: Float!) {
  patchDriftMonitor(
    input: { 
      monitorId: $monitorId, 
      set: { 
        threshold: $threshold, 
        dynamicAutoThresholdEnabled: false 
      }
    }
  ) {
    monitor { id, name, threshold }
  }
}

variables

{
  "monitorId": "monitor_id",
  "threshold": 0.25
}

Patch Performance Monitor

mutation updatePerfMonitorThreshold($monitorId: ID!, $threshold: Float!) {
  patchPerformanceMonitor(
    input: { 
      monitorId: $monitorId, 
      set: { 
        threshold: $threshold, 
        dynamicAutoThresholdEnabled: false 
      }
    }
  ) {
    monitor { id, name, threshold }
  }
}

variables

{
  "monitorId": "monitor_id"
  "threshold": 0.97
}

Each patch mutation takes a monitor's node_id an a set input that contains the properties of the monitor. Each mutation returns the monitor that has been modified - there is no need to re-query for the data! View an example Colab below.

Google Colaboratory

Patching Drift Monitors

Notice that the mutations above make use of naming a query and using variables to make the mutations re-usable.

This is a very powerful abstraction and will help you construct maintainable queries and mutations. See GraphQL docs

Deleting Monitors

Deleting monitors using a mutation simply requires you to know the ID of the monitor you want to delete.

mutation {
  deleteMonitor(input: {monitorId: "monitor_id"}) {
    clientMutationId
  }
}

Muting Monitors

You can mute a monitor by using the generic patchMonitor or a specific mutation like patchDriftMonitor, patchPerformanceMonitor, or patchDataQualityMonitor.

Generic mutation

mutation {
  patchMonitor(
    input: {
      monitorId: "monitor_id"
      set: {
        notificationsEnabled: false
      }
    }) {
    clientMutationId
  }
}

Specific mutation

mutation {
  patchMonitor(
    input: {
      monitorId: "monitor_id"
      set: {
        notificationsEnabled: false
      }
    }) {
    clientMutationId
  }
}

Creating Monitors

In most cases, the Arize platform will provide you with the best experience for creating monitors as the UI gives you a live preview of your monitor as you fill out its configuration. Additionally, Arize's managed monitors, auto-thresholding capabilities, and model baselines can often provide great monitoring coverage for your models. However, there may be cases where you would like to create a known set of custom monitors that track a specific cohort (filtered dataset) of features or a set of monitors that track a specific tag. Programmatically creating monitors can provide you the flexibility you need so you can ensure sufficient monitoring coverage. Let's suppose you have a lending model and you would like to exhaustively monitor the performance of the model for all 50 states. You may want to create a re-usable mutation that looks something like

mutation createPerformanceMonitorForState($state: String!){
  createPerformanceMonitor(
    input: {
      modelId: "model_id", 
      operator: lessThan, 
      performanceMetric: accuracy, 
      dynamicAutoThreshold: {
        stdDevMultiplier: 1.2
      }
      filters: [{
        dimensionType: featureLabel,
        operator: equals
        name: "stateName"
        values: [$state]
      }],
      contacts: [{
        notificationChannelType: email,
        emailAddress: "[e-mail]"
      }]
    }
  ) {
    monitor {
      id
    }
  }
}

Notice that we have used a variable to pass in a state to the createPerformanceMonitor mutation. We now can loop over all 50 states, invoke this mutation and get 100% coverage for all 50 states. The end result will be 50 monitors that will alert you when the performance dips below the specified threshold.

Thresholding

Two different variants of thresholds are supported: fixed thresholds and dynamic auto thresholds. Fixed thresholds are a great fit when you explicitly know when you would like the monitor to fire. However, if you would like the monitor to automatically calculate a threshold for you, you can supply an auto-threshold configuration. Note that by default, auto thresholds are enabled with a standard deviation multiplier of 2.

Auto Threshold

dynamicAutoThreshold: {
   stdDevMultiplier: 2
}

Manual Threshold

threshold: 0.01

This configuration will set your threshold to mean +/- (stdDevMultiplier * stdDev) of a historical sample. The +/- part of the threshold calculation is determined by the operator.

Having trouble? Reach out to us via email [e-mail] or Slack us in the #arize-support channel for more support.

Monitor Filters

The filters list lets you filter down the metric you are monitoring down to a cohort of inferences. It lets you filter down the data you are monitoring by features, tags, and actuals. Here are some examples:

Features and Tags

filters: [
    {
        dimensionType: featureLabel,
        name: "stateName"
        operator: equals
        values: ["california"]
    }, 
    {
        dimensionType: tagLabel,
        name: "business_impact_score"
        operator: greaterThan
        values: ["2"]
    }
]

Predictions

filters: [
    {
        dimensionType: predictionClass,
        operator: equals
        values: ["fraud"]
    }, 
    {
        dimensionType: predictionScore,
        operator: lessThan
        values: ["0.7"]
    }
]

Actuals

filters: [
    {
        dimensionType: actualClass,
        operator: equals
        values: ["fraud"]
    }, 
    {
        dimensionType: actualScore,
        operator: lessThan
        values: ["0.7"]
    }
]

GraphQL Type

"""Filters for a particular model's inference data"""
input DimensionFilterInput {
  """The part of the model to filter on."""
  dimensionType: DimensionFilterType!

  """The operator of the filter."""
  operator: ComparisonOperator!

  """Required when specifying a tag or a feature label filter."""
  name: String

  """The values to match on."""
  values: [String!]!
}

Note that numeric filter values are passed in as strings but they must be parsable as numbers.

Drift Monitor Baselines

Drift is monitored by comparing your model's current distribution to a baseline distribution. For more information on baselines and how to choose a baseline, refer to our documentation here. There are a few ways to set a baseline using our API:

Model Baseline

A model baseline refers to the baseline used for all monitors in a model. Your model is preset with a baseline defined by a moving time range of your model's production data for 17 days.

mutation {
  createDriftMonitor(input: {
    modelId: "model_id"
    dimensionCategory: predictionClass
    name: "Drift Monitor using Moving Custom Baseline"
    operator: greaterThan
    driftMetric: psi
    baseline: {
        useModelPrimaryBaseline: true,
    }
  }) {
    monitor {
      uri
    }
  }
}

Custom Moving Baseline

Custom baselines can be set for individual monitors separate of the overall model baseline. In this case, the baseline utilizes a moving time window.

mutation {
  createDriftMonitor(input: {
    modelId: "model_id"
    dimensionCategory: predictionClass
    name: "Drift Monitor using Moving Custom Baseline"
    operator: greaterThan
    driftMetric: psi
    baseline: {
        useModelPrimaryBaseline: false,
        movingWindowSeconds: 604800, // 1 week
        movingWindowDelaySeconds: 864000, // 10 days
    }
  }) {
    monitor {
      uri
    }
  }
}

Custom Fixed Baseline

Custom baselines can be set for individual monitors separate of the overall model baseline. In this case, the baseline utilizes a fixed time window.

mutation {
  createDriftMonitor(input: {
    modelId: "model_id"
    dimensionCategory: predictionClass
    name: "Drift Monitor using Fixed Custom Baseline"
    operator: greaterThan
    driftMetric: psi
    baseline: {
        useModelPrimaryBaseline: false,
        fixedStartDate: "2020-12-23T04:59:59.999Z",
        fixedEndDate: "2021-12-23T04:59:59.999Z",
    }
  }) {
    monitor {
      uri
    }
  }
}

Google Colaboratory

Creating Performance Monitors

Querying for Monitor Audit Log

The monitor audit log details all changes made to a monitor, including who made the change, the timestamp of the change, and specifics of what was altered. Use this query to track modifications and maintain oversight of monitor configurations.

query getAuditLog($monitorId: ID!) {
  node(id: $monitorId) {
    ... on Monitor {
      id
      auditLog {
        edges {
          node {
            id
            user {
              id
              name
              email
            }
            timestamp
            changeSet {
              threshold
              stdDevMultiplier
              dynamicAutoThresholdEnabled
              operator
              downtimeStart
              downtimeDurationHrs
              downtimeFrequencyDays
              name
              notes
              contacts {
                monitorId
                emailAddress
                notificationChannelType
                integration {
                  name
                  providerName
                  alertSeverity
                  channelName
                }
              }
              metric
              isNormalized
              evaluationIntervalSeconds
              topKPercentileValue
              metricAtRankingKValue
              positiveClass
              customMetric {
                metric
                name
              }
              primaryWindowModelName
              primaryWindowModelEnvironmentName
              primaryWindowFilters {
                filterType
                operator
                dimension {
                  name
                }
                dimensionValues {
                  value
                }
              }
              comparisonWindowLengthMs
              comparisonWindowStartDelaySeconds
              comparisonWindowCustomBaseline {
                modelEnvironmentName
                filteredMovingWindowSeconds
                filteredMovingWindowDelaySeconds
                filteredStartDate
                filteredEndDate
              }
              comparisonWindowDimension {
                name
              }
              comparisonWindowModelName
            }
          }
        }
      }
    }
  }
}

Querying for Integrations

Integrations belong to a Model and thus also belong to a Space. Because of this, you can either query for monitors off of a Model node or a Space node.

The model or space id is in the url when you visit the corresponding page. The format looks like this: spaces/:space_id/models/:model_id.

You can also query for a list of models given the space id. More details here.

Query by Model

query {
  node(id: "model_id") {
    ... on Model {
      monitors(first: 50) {
        edges {
          node {
            id
            name
            monitorCategory
          }
        }
      }
    }
  }
}

Query by Space

query {
  node(id: "space_id") {
    ... on Space {
      monitors(first: 50) {
        edges {
          node {
            id
            name
            monitorCategory
          }
        }
      }
    }
  }
}

Monitoring Alerts

Monitors surface potential issues with performance and data, sending real-time alerts so you can take immediate action. Learn how to programmatically update email alerts and integrate with other altering systems.

Setting Monitor Alerting Contacts

You can update either an email or paging service your monitor alerts by using the generic patchMonitor or a specific mutation such as: patchDriftMonitor, patchPerformanceMonitor, or patchDataQualityMonitor.

To view and set up external integrations see below.

Generic mutation

mutation {
  patchMonitor(
    input: $input) {
    clientMutationId
  }
}

variables

{
  "monitorId": ":monitor_id",
  "set": {
    "contacts": [
      {
        "emailAddress": "[e-mail]",
        "notificationChannelType": "email"
      },
      {
        "integrationKeyId": ":integration_id",
        "notificationChannelType": "integration"
      }
    ]
  }
}

Alerting Integrations

Natively integrate with an alerting service to identify and resolve issues faster. Learn more about our alerting integrations here.

Querying Current Integrations

Integrations belong to an AccountOrganization, thus belonging to an Account. Because of this, you must query for integrations off of an AccountOrganization node.

You can find the account organization in the url organizations/:account_organization_id/

Query

query GetIntegrationsForOrganization {
  node(id: "account_organization_id") {
    ... on AccountOrganization {
      integrations {
        id
        providerName
        name
      }
    }
  }
}

Create

mutation CreateIntegrationKey($input: CreateIntegrationKeyInput!) {
  createIntegrationKey(input: $input) {
    integrationKey {
      providerName
      name
      id
    }
  }
}

variables

{
  "input": {
    "providerName": "provider_name",
    "accountOrganizationId": "account_organization_id",
    "serviceName": "your_custom_service_name",
    "apiKey": "api_key_from_provider"
  }
}

Patch

mutation PatchIntegrationKey($input: PatchIntegrationKeyInput!) {
  patchIntegrationKey(input: $input) {
    integrationKey {
      providerName
      name
      id
    }
  }
}

variables

{
  "input": {
    "integrationKeyId": "integration_key_id",
    "set": {
      "serviceName": "new custom service name",
      "apiKey": "new_api_key_from_provider"
    }
  }
}

Delete

mutation DeleteIntegrationKey {
  deleteIntegrationKey(input: {integrationKeyId: "integration_key_to_delete"}) {
    integrationKey {
      providerName
      name
      id
    }
  }
}

Test Integration

mutation {
  testIntegrationKey(
    input: {
      integrationKeyId: "integration_key_id"
    }
  ) {
    statusCode
  } 
}

Examples

API Use Cases	Example Colabs
Patching Drift Monitors	Colab
Creating Performance Monitors	Colab
Creating Data Quality Monitors	Colab
Creating Drift Monitors	Colab