Code Evaluations

Overview

For some use cases, code-based evaluations might be a better fit than LLM as a Judge. These could be for cost reasons, latency reasons, or even the need for highly deterministic/performant responses. For these cases, there is support for setting up code based evaluations.

Guide

To create a code evaluator, follow the same steps outline in the Online Evaluations (Tasks) section. This time, select Code Evaluator instead of LLM Evaluator.

Select Code Evaluator on the New Task slide-over.

Once Code Evaluator is selected, the evaluator can be created in just 3 steps:

1. Select a manage code evaluator from the drop-down on the top right of the Code Evaluators column.

2. Provide a unique Eval Column Name for the evaluator in plaintext. Ensure that this name is distinct from other evaluators across all tasks.

3. Provide arguments to be passed into the parameters of the evaluator class.

Example where the user selects the Contains any Keyword evaluator from the drop-down, pre-populating the code block with the ContainsAnyKeyword class. The user adds the argument ["expedia", "priceline", "airbnb"] to the keywords parameter, which means the evaluator will flag any LLM output that mentions a competitor's name in the travel industry.

Managed Code Evaluators

Arize manages a set of off-the-shelf code evaluators on your behalf. Simply select the evaluator name from a drop-down and the evaluator code will be provided. Users can customize the evaluators by specifying the arguments that should be passed in as parameters. Currently, we support all of the evaluators below and new evaluators can be added upon request.

Matches Regex

This evaluator checks whether the text matches a specified regex pattern. For example, the evaluator can be used to determine whether a competitor's name is included in an LLM response to a customer. Parameters include:

• span attribute: The validation check will be applied to the content in the span attribute listed. For example, if the span attribute is attributes.llm.output, then the regex operation will apply to the LLM response. Refer to the Attributes tab of your spans on the LLM Tracing page for a list of available attributes and their content.

• pattern: The compiled regex pattern used for matching against the span attribute value.

JSON Parseable

This evaluator checks whether the LLM data is a valid JSON-parsable string. For example, this evaluator can validate that the output of an LLM can be parsed as JSON, which is a common requirement for structured data processing. Parameters include:

• span attribute: The validation check will be applied to the content in the span attribute listed. For example, if the span attribute is attributes.llm.output, then the json parseable operation will apply to the LLM response. Refer to the Attributes tab of your spans on the LLM Tracing page for a list of available attributes and their content.

Contains any Keyword

This evaluator checks whether any specified keywords are present in the LLM data. This evaluator is useful for identifying if the output contains specific terms or phrases of interest, enabling targeted validation or analysis. Parameters include:

• span attribute: The validation check will be applied to the content in the span attribute listed. For example, if the span attribute is attributes.llm.output, then the contains keyword check will apply to the LLM response. Refer to the Attributes tab of your spans on the LLM Tracing page for a list of available attributes and their content.

• keywords: A list of keyword strings to search for in the span attribute. If any keyword matches, then the evaluator will flag the data as a match.

Contains all Keywords

This evaluator is similar to the one above, except it checks that all keywords are present, rather than any. Parameters are the same as Contains any Keyword above.

Custom Code Evaluators

Coming soon! Users will be able to define imports, install packages and write custom code inheriting from the Evaluator API. We plan to support both Python and JavaScript code.

User writes a custom code evaluator class.