Test LLM Evaluator in Playground

Best Practices

The Prompt Playground is a powerful tool for testing LLM Evaluators before deploying them as an online task. Within this environment, users can easily iterate and improve their evaluator configurations:

• The Prompt Template: Experiment with different prompt structures to see which works best. For example, if you're iterating on a hallucination evaluator template, you might experiment with adding few shot examples.

• The LLM Model: Compare how different LLM models, such as GPT 4o-mini or o1-mini, affect evaluation results. You can also explore performance across various providers (e.g., Anthropic's Bedrock models or Vertex AI Gemini) and adjust LLM parameters as needed.

A key best practice is to use a golden dataset to test your evaluator. This dataset should contain carefully selected examples with ground truth labels. By comparing your evaluator’s output against these labels, you can ensure that the evaluator is performing as expected, especially in nuanced scenarios.

Workflow

Load Evaluator Template & Spans into Playground

When setting up an online task, you can use the "Test in Playground" button to evaluate your evaluator in real-time. Before entering the playground, you'll have the opportunity to preview the relevant span attributes and choose which spans from your project should be included for testing.

Select"Test in Evaluator Playground" to test the evaluator before deploying it as a task.

Preview the relevant span attributes and decide which spans from the project should be brought into the playground for testing.

Iterate on Evaluator to Generate New Output

Once inside the playground, select "Run" to generate output based on the selected spans. When using our provided function, this output will include both an evaluation label (e.g., "factual" or "hallucinated") and an explanation to clarify the reasoning behind the label.

If the evaluation label is incorrect, consider iterating on your template or adjusting the model to refine the results. You can repeat this process as many times as needed to fine-tune your evaluator.

Once in the playground, select Run to generate output across the loaded spans. In this example, we can see that the first example is labeled as "factual", alongside an explanation from the LLM. If the evaluation label is incorrect, try iterating on the template.

Load in a Golden Dataset

Additionally, you can load a golden dataset consisting of carefully selected examples to test the evaluator's performance across a specific set of test cases, ensuring its accuracy and consistency.

Load in a golden dataset of carefully selected examples to get a better understanding of the LLM Evaluator performance.

Update Task with Changes

After testing in the playground and making improvements—whether by adjusting the model, modifying LLM parameters, or fine-tuning the prompt template—return to the task creation page to apply these changes. This step ensures that the updates made during testing are reflected in your live task, allowing for more accurate evaluations in production.

Select "Back to Task Creation" to return to the online eval setup page, then choose "Use Updated Prompt" to include your changes.