Quickstart: Prompt Engineering

Content

This Quickstart guide will walk you through the following content:

• Find problematic production examples and pull them into the playground for debugging.

• Improve your model and prompt template to address identified issues.

• Test the updated prompt on a dataset of examples.

• Save the updated prompt as an experiment and evaluate aggregate metrics to share with your team.

• Publish the updated prompt to the Prompt Hub to update your production application.

Overview

The Prompt Playground provides an intuitive interface for experimenting with prompt templates, input variables, LLM models, and parameters. This no-code platform empowers both coding and non-coding experts to refine their prompts for optimal use in production applications.

Load Span into Playground

The most common way to enter the Prompt Playground is through a span on the LLM tracing page. For instance, users can filter spans where an Online Evaluator flagged the LLM output as a hallucination and then bring one of these examples into the Prompt Playground to refine the prompt, ensuring the LLM produces factual responses in the future.

User applies filters to identify examples where the Online Evaluator flagged the LLM output as a hallucination.

User selects the Prompt Playground button to import the template, input variables, and LLM output for iteration in the playground.

User modifies the template and input variables in the Playground to refine and iterate on the prompt.

Iterate on Prompt in Playground

Experiment with Model & LLM Parameters

With the span loaded in the Playground, the user can iterate on the prompt to reduce the likelihood of hallucinations. One effective approach is to adjust the model and its parameters. In this example, the user reduces the model's temperature and upgrade to GPT-4o for improved accuracy and reliability.

Note that the Prompt Playground supports all major model providers and allows integration with custom model endpoints.

User upgrades the model to GPT-4o and reduces the temperature to zero.

Experiment with Template & Variables

Another approach to reducing hallucinations is modifying the template. Using Copilot, the user optimizes the prompt, instructing the LLM to respond with 'I don’t know' when the answer is not found in the provided context. After pressing 'Run' with the updated prompt template, the New Output confirms that the LLM now responds with 'I don’t know' instead of generating a fabricated answer.

User requests Copilot to optimize the prompt to reduce the likelihood of the LLM generating hallucinated responses.

Copilot suggests an optimized response, highlighted in green, which the user can review and choose to accept or reject.

User clicks 'Run' with the optimized template from Copilot, and the updated LLM output accurately responds with 'I don’t know,' eliminating hallucinated responses.

Load Dataset into Playground

While we have observed improved performance on a single example, how can we ensure consistent improvement across all hallucinated spans? To validate that the new prompt effectively reduces hallucinations more broadly, we can load a dataset of hallucinated examples into the Prompt Playground and test the updated prompt against the entire dataset.

User selects 'Load a Dataset' to import a dataset of inputs and outputs where the LLM hallucinated. In this case, the user chooses not to override the prompt template with the dataset's template, allowing them to test the new template in the Playground.

User tests the new template on all 10 examples in the dataset and confirms that the LLM responds with 'I don’t know' for 9 out of the 10 examples.

User reviews each example in the 'Compare Outputs' view, where they can examine the original and updated outputs side-by-side in full-screen mode, toggling between JSON and text formatting as needed.

Save Playground States

Save Outputs as Experiment

Having observed improvements across the entire dataset, the next step is to save the output for team review and collaboration. The output is saved as an experiment, allowing comparison with other templates. Evaluation metrics are computed and aggregated for each experiment, offering a quantitative analysis to complement the qualitative review of the new outputs.

After generating new results in the Prompt Playground, the user selects 'Save as Experiment' to organize and store the output under a specific Experiment Name for future comparison and collaboration.

After successfully creating an experiment, the user selects 'View Experiment' to navigate to the experiments page for reviewing and analyzing the saved results in detail.

The newly created 'Hallucinate Less Template from Copilot' experiment appears at the top of the experiments list, displaying its creation time and other associated details for easy reference and comparison.

By clicking into the experiment, the user can review detailed outputs alongside additional metadata, such as example IDs, experiment IDs, and the information needed to reproduce the LLM call, including invocation parameters and the template used.

The user includes additional experiments for a side-by-side comparison of outputs, streamlining the view by de-selecting metadata columns to focus on the most relevant information.

Save Template to Hub

The template can also be saved to the Prompt Hub, making it especially valuable for production use cases and collaboration. Refer to the next section for details.