Quickstart: Datasets

This guide helps you create golden datasets to benchmark your LLM application performance. Learn more about the concepts of datasets here.

You can create your dataset in four ways:

1. Import spans from your application

2. Create your dataset in code

3. Auto-generate using LLMs

4. Upload a CSV (coming soon)

Import spans from your application

If you have added tracing to your application, you can create datasets by adding spans from your application with Arize. Go to the traces page and filter for the examples you care about.

In the example below, we are filtering for spans with a hallucination label, and adding them to a dataset.

Use AI search to curate your dataset with natural language.✨

Create your dataset in code

To start let's install the packages we need:

!pip install "arize[Datasets]" pandas

Let's get your developer key by clicking "code" on the datasets page.

Let's setup the Arize Dataset Client to create or update a dataset.

from arize.experimental.datasets import ArizeDatasetsClient
import pandas as pd

client = ArizeDatasetsClient(developer_key=developer_key)

You can create many different kinds of datasets. We'll start with the simplest example below, and if you'd like to upload datasets with prompt variables, edit or delete your datasets, follow this guide.

Upload a simple dataset

import pandas as pd
import json
from arize.experimental.datasets.utils.constants import GENERATIVE

data = [{
    "persona": "An aspiring musician who is writing their own songs",
    "problem": "I often get stuck overthinking my lyrics and melodies.",
}]

df = pd.DataFrame(data)

dataset_id = client.create_dataset(
    space_id="YOUR_SPACE_ID", 
    dataset_name="Your Dataset",
    dataset_type=GENERATIVE,
    data=df
)

Auto-generate using LLMs

When you are first developing with LLMs, you typically start with a prompt and little else. The early iteration gets you to a point where the video demo looks amazing, but there's a lack of confidence in its reliability and robustness.

This is where you can use LLMs to generate examples for you based on your prompt. Here's an example, where we can use ChatGPT or your LLM of choice to create a set of examples you can upload to Arize.

You are a data analyst. You are using LLMs to summarize a document. Create a CSV of 20 test cases with the following columns:

1. Input: The full document text, usually five paragraphs of articles about beauty products.
2. Prompt Variables: A JSON string of metadata attached to the article, such as the article title, date, and website URL
3. Output: The one line summary

This will generate a CSV file for you that looks like:

Coming soon, you'll be able to do this directly in the Arize platform. You can use the auto-generated examples and upload them using code or CSV below.

Upload a CSV (coming soon)

If you are managing your existing data points today in google sheets or excel, you can import those directly into Arize. You can set your input variables and expected output for your experiment runs.