Using Arize Evaluators

Overview

We have built simple functions for using our eval prompt templates. These prompts are tested against benchmarked datasets and target precision at 70-90% and F1 at 70-85%. We use Phoenix, our open-source package to run evaluations.

To use our evaluators, follow these steps below.

Choose an evaluator LLM Classify

The LLM Classify allows you to run LLM as a Judge. Arize has open sourced the code for LLM as a Judge in Phoenix.

Evaluator

Required Columns

Output Labels

Use

Have Copilot Choose an Evaluator

If you are unsure where eval to choose,✨Copilot can choose for you. Navigate to the main chat in the UI and ask Copilot to suggest a Phoenix eval for your application.

Using LLM_Classify

The LLM Classify uses a library to classify and generate Evals. Arize uses the Arize Phoenix open source library to run phoenix LLM as a Judge.

Setup the evaluation library

All of our evaluators are easily imported with the phoenix library, which you can install using this command below.

pip install -q "arize-phoenix>=4.29.0"
pip install -q openai

Import the pre-tested evaluators along with the helper functions using this code snippet.

import phoenix.evals
import os 
from phoenix.evals import (
    QA_PROMPT_RAILS_MAP,
    QA_PROMPT_TEMPLATE,
    HALLUCINATION_PROMPT_RAILS_MAP,
    HALLUCINATION_PROMPT_TEMPLATE,
    OpenAIModel,
    llm_classify,
)

Next, you need to setup the evaluators to use a specific large language model provider. This example uses OpenAIModel, but you can use any of our supported evaluation models. In this example, we will use the hallucination evaluator and the QA correctness evaluator.

openai_api_key = "YOUR_OPENAI_KEY"
os.environ["OPENAI_API_KEY"] = openai_api_key

eval_model = OpenAIModel(model="gpt-4o", api_key=openai_api_key)

Prepare your data

Our evaluation functions require dataframes to be passed with specific column names. You can construct these dataframes manually or you can manipulate the dataframes you retrieve from traces in Arize or traces in Phoenix.

For this example, we will create the dataframe from scratch to include the required columns we need -- input, reference, and output.

import pandas as pd
dataframe = pd.DataFrame(
    [
        {
            "input": "What is the capital of California?",
            "reference": "Sacramento is the capital of California.",
            "output": "Sacramento",
        },
        {
            "input": "What is the capital of California?",
            "reference": "Carson City is the Capital of Nevada.",
            "output": "Carson City",
        },
    ]
)

Run the eval

Then you can use the llm_classify function to run the evals on your dataframe.

rails = list(QA_PROMPT_RAILS_MAP.values())
qa_correctness_eval_df = llm_classify(
    dataframe=dataframe,
    template=QA_PROMPT_TEMPLATE,
    model=eval_model,
    rails=rails,
    concurrency=20,
)

rails = list(HALLUCINATION_PROMPT_RAILS_MAP.values())
hallucination_eval_df = llm_classify(
    dataframe=dataframe, 
    template=HALLUCINATION_PROMPT_TEMPLATE, 
    model=eval_model, 
    rails=rails,
    provide_explanation=True, #optional to generate explanations for the value produced by the eval LLM
)

Now you have the results of your hallucination eval and QA correctness eval! See the results below when you print your results (data not included).

hallucination_eval_df

qa_correctness_eval_df

If you'd like, you can log those evaluations back to Arize to save the results.

Supported Models

The models are instantiated and usable in the LLM Eval function. The models are also directly callable with strings.

model = OpenAIModel(model_name="gpt-4",temperature=0.6)
model("What is the largest costal city in France?")

We currently support a growing set of models for LLM Evals, please check out the API section for usage.

🎓 Learn more about the concept of LLM as a judge.

PreviousHow To: Evaluations NextHallucinations

Last updated 23 days ago