LogoLogo
Python SDKSlack
  • Documentation
  • Cookbooks
  • Self-Hosting
  • Release Notes
  • Reference
  • Arize AI
  • Quickstarts
  • ✨Arize Copilot
  • Arize AI for Agents
  • Concepts
    • Agent Evaluation
    • Tracing
      • What is OpenTelemetry?
      • What is OpenInference?
      • Openinference Semantic Conventions
    • Evaluation
  • 🧪Develop
    • Quickstart: Experiments
    • Datasets
      • Create a dataset
      • Update a dataset
      • Export a dataset
    • Experiments
      • Run experiments
      • Run experiments with code
        • Experiments SDK differences in AX vs Phoenix
        • Log experiment results via SDK
      • Evaluate experiments
      • Evaluate experiment with code
      • CI/CD with experiments
        • Github Action Basics
        • Gitlab CI/CD Basics
      • Download experiment
    • Prompt Playground
      • Use tool calling
      • Use image inputs
      • Replay spans
      • Compare prompts side-by-side
      • Load a dataset into playground
      • Save playground outputs as an experiment
      • ✨Copilot: prompt builder
    • Playground Integrations
      • OpenAI
      • Azure OpenAI
      • AWS Bedrock
      • VertexAI
      • Custom LLM Models
    • Prompt Hub
  • 🧠Evaluate
    • Online Evals
      • Run evaluations in the UI
      • Run evaluations with code
      • Test LLM evaluator in playground
      • View task details & logs
      • ✨Copilot: Eval Builder
      • ✨Copilot: Eval Analysis
      • ✨Copilot: RAG Analysis
    • Experiment Evals
    • LLM as a Judge
      • Custom Eval Templates
      • Arize Templates
        • Agent Tool Calling
        • Agent Tool Selection
        • Agent Parameter Extraction
        • Agent Path Convergence
        • Agent Planning
        • Agent Reflection
        • Hallucinations
        • Q&A on Retrieved Data
        • Summarization
        • Code Generation
        • Toxicity
        • AI vs Human (Groundtruth)
        • Citation
        • User Frustration
        • SQL Generation
    • Code Evaluations
    • Human Annotations
  • 🔭Observe
    • Quickstart: Tracing
    • Tracing
      • Setup tracing
      • Trace manually
        • Trace inputs and outputs
        • Trace function calls
        • Trace LLM, Retriever and Tool Spans
        • Trace prompt templates & variables
        • Trace as Inferences
        • Send Traces from Phoenix -> Arize
        • Advanced Tracing (OTEL) Examples
      • Add metadata
        • Add events, exceptions and status
        • Logging Latent Metadata
        • Add attributes, metadata and tags
        • Send data to a specific project
        • Get the current span context and tracer
      • Configure tracing options
        • Configure OTEL tracer
        • Mask span attributes
        • Redact sensitive data from traces
        • Instrument with OpenInference helpers
      • Query traces
        • Filter Traces
          • Time Filtering
        • Export Traces
        • ✨AI Powered Search & Filter
        • ✨AI Powered Trace Analysis
        • ✨AI Span Analysis & Evaluation
    • Tracing Integrations
      • OpenAI
      • OpenAI Agents SDK
      • LlamaIndex
      • LlamaIndex Workflows
      • LangChain
      • LangGraph
      • Hugging Face smolagents
      • Autogen
      • Google GenAI (Gemini)
      • Model Context Protocol (MCP)
      • Vertex AI
      • Amazon Bedrock
      • Amazon Bedrock Agents
      • MistralAI
      • Anthropic
      • LangFlow
      • Haystack
      • LiteLLM
      • CrewAI
      • Groq
      • DSPy
      • Guardrails AI
      • Prompt flow
      • Vercel AI SDK
      • Llama
      • Together AI
      • OpenTelemetry (arize-otel)
      • BeeAI
    • Evals on Traces
    • Guardrails
    • Sessions
    • Dashboards
      • Dashboard Widgets
      • Tracking Token Usage
      • ✨Copilot: Dashboard Widget Creation
    • Monitors
      • Integrations: Monitors
        • Slack
          • Manual Setup
        • OpsGenie
        • PagerDuty
      • LLM Red Teaming
    • Custom Metrics & Analytics
      • Arize Query Language Syntax
        • Conditionals and Filters
        • All Operators
        • All Functions
      • Custom Metric Examples
      • ✨Copilot: ArizeQL Generator
  • 📈Machine Learning
    • Machine Learning
      • User Guide: ML
      • Quickstart: ML
      • Concepts: ML
        • What Is A Model Schema
        • Delayed Actuals and Tags
        • ML Glossary
      • How To: ML
        • Upload Data to Arize
          • Pandas SDK Example
          • Local File Upload
            • File Upload FAQ
          • Table Ingestion Tuning
          • Wildcard Paths for Cloud Storage
          • Troubleshoot Data Upload
          • Sending Data FAQ
        • Monitors
          • ML Monitor Types
          • Configure Monitors
            • Notifications Providers
          • Programmatically Create Monitors
          • Best Practices for Monitors
        • Dashboards
          • Dashboard Widgets
          • Dashboard Templates
            • Model Performance
            • Pre-Production Performance
            • Feature Analysis
            • Drift
          • Programmatically Create Dashboards
        • Performance Tracing
          • Time Filtering
          • ✨Copilot: Performance Insights
        • Drift Tracing
          • ✨Copilot: Drift Insights
          • Data Distribution Visualization
          • Embeddings for Tabular Data (Multivariate Drift)
        • Custom Metrics
          • Arize Query Language Syntax
            • Conditionals and Filters
            • All Operators
            • All Functions
          • Custom Metric Examples
          • Custom Metrics Query Language
          • ✨Copilot: ArizeQL Generator
        • Troubleshoot Data Quality
          • ✨Copilot: Data Quality Insights
        • Explainability
          • Interpreting & Analyzing Feature Importance Values
          • SHAP
          • Surrogate Model
          • Explainability FAQ
          • Model Explainability
        • Bias Tracing (Fairness)
        • Export Data to Notebook
        • Automate Model Retraining
        • ML FAQ
      • Use Cases: ML
        • Binary Classification
          • Fraud
          • Insurance
        • Multi-Class Classification
        • Regression
          • Lending
          • Customer Lifetime Value
          • Click-Through Rate
        • Timeseries Forecasting
          • Demand Forecasting
          • Churn Forecasting
        • Ranking
          • Collaborative Filtering
          • Search Ranking
        • Natural Language Processing (NLP)
        • Common Industry Use Cases
      • Integrations: ML
        • Google BigQuery
          • GBQ Views
          • Google BigQuery FAQ
        • Snowflake
          • Snowflake Permissions Configuration
        • Databricks
        • Google Cloud Storage (GCS)
        • Azure Blob Storage
        • AWS S3
          • Private Image Link Access Via AWS S3
        • Kafka
        • Airflow Retrain
        • Amazon EventBridge Retrain
        • MLOps Partners
          • Algorithmia
          • Anyscale
          • Azure & Databricks
          • BentoML
          • CML (DVC)
          • Deepnote
          • Feast
          • Google Cloud ML
          • Hugging Face
          • LangChain 🦜🔗
          • MLflow
          • Neptune
          • Paperspace
          • PySpark
          • Ray Serve (Anyscale)
          • SageMaker
            • Batch
            • RealTime
            • Notebook Instance with Greater than 20GB of Data
          • Spell
          • UbiOps
          • Weights & Biases
      • API Reference: ML
        • Python SDK
          • Pandas Batch Logging
            • Client
            • log
            • Schema
            • TypedColumns
            • EmbeddingColumnNames
            • ObjectDetectionColumnNames
            • PromptTemplateColumnNames
            • LLMConfigColumnNames
            • LLMRunMetadataColumnNames
            • NLP_Metrics
            • AutoEmbeddings
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
          • Single Record Logging
            • Client
            • log
            • TypedValue
            • Ranking
            • Multi-Class
            • Object Detection
            • Embedding
            • LLMRunMetadata
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
        • Java SDK
          • Constructor
          • log
          • bulkLog
          • logValidationRecords
          • logTrainingRecords
        • R SDK
          • Client$new()
          • Client$log()
        • Rest API
    • Computer Vision
      • How to: CV
        • Generate Embeddings
          • How to Generate Your Own Embedding
          • Let Arize Generate Your Embeddings
        • Embedding & Cluster Analyzer
        • ✨Copilot: Embedding Summarization
        • Similarity Search
        • Embedding Drift
        • Embeddings FAQ
      • Integrations: CV
      • Use Cases: CV
        • Image Classification
        • Image Segmentation
        • Object Detection
      • API Reference: CV
Powered by GitBook

Support

  • Chat Us On Slack
  • support@arize.com

Get Started

  • Signup For Free
  • Book A Demo

Copyright © 2025 Arize AI, Inc

On this page
  • Quickstart Guide
  • Managed Code Evaluators
  • Custom Code Evaluators

Was this helpful?

  1. Evaluate

Code Evaluations

Run python code as background tasks against your span data

Last updated 14 days ago

Was this helpful?

When your evaluation criteria is deterministic, such as evaluating the existence of a keyword or URL within the output text, it is best to use code-based evaluations.

Quickstart Guide

Once Code Evaluator is selected, the evaluator can be created in just 3 steps:

  1. Select a manage code evaluator from the drop-down on the top right of the Code Evaluators column.

  2. Provide a unique Eval Column Name for the evaluator in plaintext. Ensure that this name is distinct from other evaluators across all tasks.

  3. Provide arguments to be passed into the parameters of the evaluator class.

Managed Code Evaluators

Arize manages a set of off-the-shelf code evaluators on your behalf. Simply select the evaluator name from a drop-down and the evaluator code will be provided. Users can customize the evaluators by specifying the arguments that should be passed in as parameters. Currently, we support all of the evaluators below and new evaluators can be added upon request.

Matches Regex

This evaluator checks whether the text matches a specified regex pattern. For example, the evaluator can be used to determine whether a competitor's name is included in an LLM response to a customer. Parameters include:

  • span attribute: The validation check will be applied to the content in the span attribute listed. For example, if the span attribute is attributes.llm.output, then the regex operation will apply to the LLM response. Refer to the Attributes tab of your spans on the LLM Tracing page for a list of available attributes and their content.

  • pattern: The compiled regex pattern used for matching against the span attribute value.

JSON Parseable

This evaluator checks whether the LLM data is a valid JSON-parsable string. For example, this evaluator can validate that the output of an LLM can be parsed as JSON, which is a common requirement for structured data processing. Parameters include:

  • span attribute: The validation check will be applied to the content in the span attribute listed. For example, if the span attribute is attributes.llm.output, then the json parseable operation will apply to the LLM response. Refer to the Attributes tab of your spans on the LLM Tracing page for a list of available attributes and their content.

Contains any Keyword

This evaluator checks whether any specified keywords are present in the LLM data. This evaluator is useful for identifying if the output contains specific terms or phrases of interest, enabling targeted validation or analysis. Parameters include:

  • span attribute: The validation check will be applied to the content in the span attribute listed. For example, if the span attribute is attributes.llm.output, then the contains keyword check will apply to the LLM response. Refer to the Attributes tab of your spans on the LLM Tracing page for a list of available attributes and their content.

  • keywords: A list of keyword strings to search for in the span attribute. If any keyword matches, then the evaluator will flag the data as a match.

Contains all Keywords

This evaluator is similar to the one above, except it checks that all keywords are present, rather than any. Parameters are the same as Contains any Keyword above.

Custom Code Evaluators

Custom Code Evals allow you to define your own evaluation logic in Python (with JavaScript coming soon) to score and label LLM traces based on span attributes. This is ideal for use cases that require highly customized and deterministic rules—such as business logic validation, structured output parsing, or expected keyword presence.

Once you select CustomArizeEvaluator from the "Select an Eval" drop-down, you’ll define the logic in the right-hand panel of the task creation interface.

Step 1: Imports

Start by importing the necessary classes and functions you'll need in your evaluator. This is easiest to do in full screen view, using the expand button on the top right of each code cell.

Currently we support the packages listed below. If you need an additional package installed, please notify the customer support team and we'll do our best to address your requirements!

numpy
pandas
scipy
pyarrow
arize[Datasets]==7.25.7

Step 2: Test in Code

While it's possible to write the code in the UI, it's typically easier to iterate in a python script or Colab notebook. We provide the necessary starter code with a Test in Code button. The starter code will port over the span attributes, evaluator class and import sections from your task, with buttons to copy the code snippets and run the code over your own data.

Step 3: Update Evaluator Class and Span Attributes

To create a code evaluator, follow the same steps outline when . This time, select Code Evaluator instead of LLM Evaluator.

Once you're seeing the desired results with your evaluation code, we recommend copy-pasting the updated child class into the UI, along with the span attributes and imports, if needed. You're now ready to kick off your task!

🧠
running evaluations in the UI
CodeEvaluator
Select Code Evaluator on the New Task slide-over.
Example where the user selects the Contains any Keyword evaluator from the drop-down, pre-populating the code block with the ContainsAnyKeyword class. The user adds the argument ["expedia", "priceline", "airbnb"] to the keywords parameter, which means the evaluator will flag any LLM output that mentions a competitor's name in the travel industry.