Quickstart: Evaluation

Overview

This guide will show you how to setup an online evaluation in the Arize UI, which runs on your data automatically as your LLM application is used. You can setup LLM as a judge evaluators or code evaluators, which run in a python container against your data.

If you'd like to run your own evaluations offline using the Arize SDK, click here to follow this guide.

Step 1: Create a new task

Navigate to the tasks page and click "new task".

Select New Task on the Tasks page.

Step 2: Select your data

Choose which traces you want to evaluate, and how often you want to run it. You can run it once to fill historical data, or you can run it continuously against new data.

Fill out the New Task column on the left.

Step 3: Choose your LLM

Choose the LLM provider, model, and other parameters for your LLM evaluation.

Define the LLM Parameters.

Step 4: Define your evaluator

Select one of our pre-built evaluators, or use copilot to write one on your behalf! You can also write your own evaluation template or write python code to evaluate your LLM outputs.

Define the LLM Evaluator.

Use Copilot to write your evaluator!

Step 5. View Evaluation Labels on LLM Tracing Page

Once your task is successfully created, a green pop-up notification will appear! Navigate to the tracing page to view the evaluation labels generated by your newly created task.

New task appears on the Tasks page. Green pop up provides a link to the LLM Tracing page, where you can see your evaluation labels.

View labels in the Evaluations column of the LLM Tracing page associated with the project selected in the task.