Custom Metrics

Learn how to query, create, and update custom metrics programmatically

Use custom metrics to define a metric specific to your business use case or to your model. By using a programmatic API, you can automate the process of creating a custom metric, updating that metric, and creating monitors on that metric.

Learn more about custom metrics here.

For a brief overview of GraphQL itself, please consult our introduction.

Creating a custom metric

Creating a custom metric directly:

mutation {
  createCustomMetric(
    input: { 
      name: "prediction count", 
      description: "Returns the count of predictions over time", 
      modelId: "model_id", 
      metric: "SELECT COUNT(*) FROM model" 
  }) {
    customMetric {
      id
    }
  }
}

A more repeatable way to do this is to use query parameters:

# Query:
mutation myCustomMetricMutation($input: CreateCustomMetricMutationInput!) {
  createCustomMetric(input: $input) {
    customMetric {
      id
    }
  }
}

# Params:
{
    "input": { 
        "name": "prediction count", 
        "description": "Returns the count of predictions over time", 
        "modelId": "model_id", 
        "metric": "SELECT COUNT(*) FROM model"
    }
}

Note that error messages work the same in GraphQL as in the UI. For example, if my metric was this: SELECT COUNT(income) FROM model, I would get this error message in both the UI and the API:

{
  "errors": [
    {
      "message": "The custom metric failed to save: column \"income\" does not exist on line 1, column 13",
      ...

However, due to the ability to use auto complete and the model schema panel in the UI, we do encourage using the UI to create an initial custom metric before attempting to create one directly using the GraphQL API.

Updating a custom metric

In order to update a custom metric, use the updateCustomMetricMutation. Follow an example here.

mutation {
  updateCustomMetric(input: {
      customMetricId:"custom_metric_id",
      modelId:"model_id",
      name:"new custom metric name",
      description:"new description",
      customMetric:"SELECT avg(annual_income) FROM model"
  }) {    
    customMetric {
      id
    }
  }
}

Querying a custom metric

You can query for custom metrics associated with a model like this:

{
  node(id: "model_id") {
    ... on Model {
      customMetrics(first: 10) {
        edges {
          node {
            name
            metric
            description
          }
        }
      }
    }
  }
}

If you know the id of your custom metric - you can also query for it directly:

{
  node(id:"custom_metric_id"){
    ... on CustomMetric {
      id
      name
      description
      metric
    }
  }
}

Creating a monitor for a custom metric

Users can monitor a custom metric using the performance monitor type. Follow an example here.

To create a custom metric monitor - first get the custom metric id. This can come from the result of the createCustomMetricMutation above, or it can come from the URL of the custom metric:

app.arize.com/organizations/:org_id/spaces/:space_id/models/:model_id/custom_metrics/:custom_metric_id/

Then, use the createPerformanceMonitorMutation to create the monitor:

mutation {
  createPerformanceMonitor(input:{
     modelId: "model_id",
     customMetricId: "custom_metric_id",
     operator: greaterThan,
     name: "custom metric monitor"
  }){
    monitor{
      id
      uri
    }
  }
}

Examples

API Use Cases	Example Colabs
Create Custom Metrics With GraphQL	Colab
Update Custom Metrics With GraphQL	Colab

To learn more about performance monitors - refer to our monitors API documentation here.