How to: Experiments

Experiments are offline tests of your system that consist of tasks for running your test, a dataset for running the test against and an evaluator for checking the results.

Create a Task

Create an Experiment Evaluator

Code Based Eval Experiment

LLM Eval Experiment

Evaluator Input and Return Types Supported

Trace an Experiment

Download Experiment Results

Run Async Vs Sync Tasks and Evals

Create an Experiment Evaluator

Sample a Dataset for an Experiment

CI/CD for Automated Experiments