Overview: LLM Evaluation

Grading the performance of your LLM application

An evaluation example for a Q&A chatbot in Arize

The purpose of evaluation

In order to ensure that your application performs well, you need a way to judge the quality of your LLM outputs. You can evaluate many different factors, such as relevance, hallucination %, and latency.

Without evals, AI engineers don’t know if prompt changes will actually improve performance. They don't know whether changing a prompt, LLM parameter or retrieval step, will it break a use case or actually improve performance?

With evals, when you adjust your prompts, agents, and retrieval, you will know whether your application has improved.

Once you have evaluation metrics, you need a good evaluation process. You need a set of examples that you've curated to test against a variety of use cases, and run those tests on every build in your CI/CD pipeline. Now you can constantly test your application for reliability on every change.

It's important to select a comprehensive dataset which will determine how trustworthy and generalizable your evaluation metrics are to production use. A limited dataset could showcase high scores on evaluation metrics, but perform poorly in real-world scenarios.

Learn more

Types of LLM evals

To get started with code, check out the Quickstart guide for LLM evaluation.

Learn more about evaluation concepts by reading Evaluation Basics and our definitive guide on LLM app evaluation.

Trying to setup online evaluations that run continuously on your data? See our online evaluations guide.