[COMING SOON] Evaluate Voice Applications

Currently, voice evaluations are supported exclusively with OpenAI models. Support for additional models is planned and will be available soon.

This guide provides instructions on how to evaluate voice applications using OpenAI models within the Phoenix framework. The example notebook linked below demonstrates the process of configuring and running evaluations.

Prerequisites

1. Phoenix Installation: Make sure the phoenix package is installed in your Python environment.

2. OpenAI API Key: Obtain an API key for the OpenAI model you plan to use.

3. Audio Data: Prepare the audio data required for evaluation. This can be in the form of raw audio bytes, base64-encoded strings, or URLs pointing to audio files. If you have existing data in Arize, you can use our export client to retrieve it.

4. Python Environment: Ensure you are using Python version 3.7 or higher.

Steps to Evaluate

1. Set Up the Model

Use the OpenAIModel class to define the OpenAI model for evaluation. Replace the placeholder API key with your own.

from phoenix.evals import OpenAIModel

model = OpenAIModel(model="gpt-4o-audio-preview", api_key="your_openai_api_key")

2. Define the Template

Templates are used to configure the prompts sent to the OpenAI model. They consist of "rails" (the set of valid responses) and a sequence of prompt parts, each specifying the type and content of the input or instructions. Below is a complete example for a tone and emotion classification template.

Explanation of Components

• Rails: These are the valid outputs for the classification task. In this case, the rails are ["positive", "neutral", "negative"].

• Prompt Parts: The template breaks the prompt into multiple parts, each of a specific content type:

  • TEXT: Static instructions or context provided as text.

  • AUDIO: Dynamically inserted audio data for analysis.

• Templates: These define the content that will be sent to the model. For example, one part provides a description of the task, while another includes the audio data.

from phoenix.evals.templates import (
    ClassificationTemplate,
    PromptPartContentType,
    PromptPartTemplate,
)

# Define valid classification labels (rails)
TONE_EMOTION_RAILS = ["positive", "neutral", "negative"]

# Create the classification template
template = ClassificationTemplate(
    rails=TONE_EMOTION_RAILS,  # Specify the valid output labels
    template=[
        # Prompt part 1: Task description
        PromptPartTemplate(
            content_type=PromptPartContentType.TEXT,
            template="""
            You are a helpful AI bot that checks for the emotional sentiment of the audio.
            Analyze the audio file and determine the sentiment (e.g., positive, neutral, negative).
            Your evaluation should provide a multiclass label from the following options: ['positive', 'neutral', 'negative'].
            
            Here is the audio:
            """,
        ),
        # Prompt part 2: Insert the audio data
        PromptPartTemplate(
            content_type=PromptPartContentType.AUDIO,
            template="{attributes.input.audio.url}",  # Placeholder for the audio content
        ),
        # Prompt part 3: Define the response format
        PromptPartTemplate(
            content_type=PromptPartContentType.TEXT,
            template="""
            Your response must be a string, either positive, neutral, or negative, and should not contain any text or characters aside from that.
            """,
        ),
    ],
)

How It Works

1. Prompt Parts:

   • The first part provides clear instructions to the model, explaining the task and the valid response labels.

   • The second part dynamically inserts the audio data for evaluation.

   • The third part ensures the model outputs the result in the required format (a single-word string with no extra characters).

2. Rails:

   • These define the set of valid responses that the model can produce (i.e positive, neutral, negative). Any result outside this set can be flagged or considered invalid.

3. Dynamic Inputs:

   • The {audio_data} placeholder is replaced with the actual base64-encoded audio string at runtime. This is handled by llm_classify when the content_type is set to PromptPartContentType.AUDIO. The variable name should correspond with the column containing either your audio url or data

   • If you only want to evaluate text (i.e the transcript) then you can just use the prompt directly as a string.

This modular approach ensures flexibility, allowing you to reuse and adapt templates for different use cases or models.

3. Prepare the Data Processor (Optional)

A data processor transforms raw audio data into base64-encoded strings. For example, fetching and encoding audio from Google Cloud Storage:

import asyncio
import base64
import aiohttp

async def async_fetch_gcloud_data(url: str) -> str:
    token = None
    try:
        # Execute the gcloud command to fetch the access token
        output = await asyncio.create_subprocess_exec(
            "gcloud",
            "auth",
            "print-access-token",
            stdout=asyncio.subprocess.PIPE,
            stderr=asyncio.subprocess.PIPE,
        )
        stdout, stderr = await output.communicate()
        if output.returncode != 0:
            raise RuntimeError(f"Error executing gcloud command: {stderr.decode('UTF-8').strip()}")
        token = stdout.decode("UTF-8").strip()

        # Ensure the token is not empty or None
        if not token:
            raise ValueError("Failed to retrieve a valid access token. Token is empty.")

    except Exception as e:
        # Catch any other exceptions and re-raise them with additional context
        raise RuntimeError(f"An unexpected error occurred: {str(e)}")

    # Set the token in the header
    gcloud_header = {"Authorization": f"Bearer {token}"}

    # Must ensure that the url begins with storage.googleapis..., rather than store.cloud.google...
    G_API_HOST = "https://storage.googleapis.com/"
    not_googleapis = url.startswith("https://storage.cloud.google.com/") or url.startswith("gs://")
    g_api_url = (
        url.replace("https://storage.cloud.google.com/", G_API_HOST)
        if url and not_googleapis
        else url
    )

    # Get a response back, present the status
    async with aiohttp.ClientSession() as session:
        async with session.get(g_api_url, headers=gcloud_header) as response:
            response.raise_for_status()
            content = await response.read()

    encoded_string = base64.b64encode(content).decode("utf-8")

    return encoded_string

If your audio data is already in base64 format as an encoded string, you can skip this step.

4. Perform the Evaluation

Use the llm_classify function to run the evaluation. Pass your data as a Pandas DataFrame.

from phoenix.evals.classify import llm_classify
import pandas as pd

# Example DataFrame
df = client.export_model_to_df(
    space_id='SPACE_ID',
    model_id='PROJECT_NAME',
    environment=Environments.TRACING,
    start_time=datetime.fromisoformat('2024-12-23T07:00:00.000+00:00'),
    end_time=datetime.fromisoformat('2024-12-31T06:59:59.999+00:00'),
    )

# Run the evaluation
results = llm_classify(
    dataframe=df,
    data_processor=async_fetch_gcloud_data,  # Optional, depending on your data format
    model=model,
    template=template,
    rails=TONE_EMOTION_RAILS,
    provide_explanation=False,  # Set to True if explanations are needed
)

Considerations

• Data Processors: Custom functions can transform audio paths, URLs, or raw data to the required format.

• Templates: Modify the templates to fit your specific evaluation needs.

  • note: template variables should have the same naming as the desired columns in the dataframe

• Explanations: Enable provide_explanation=True to include detailed reasoning in the evaluation output.