QA with Retrieval (Using Vector Stores)

Helps answer questions such as: Are there queries that don’t have sufficient context? Should you add more context for these queries to get better answers? Or can you change your embeddings?

Framework	Phoenix Inferences	Phoenix Traces & Spans
LangChain	Retrieval Analyzer w/ Embeddings	Traces and Spans
LlamaIndex	Retrieval Analyzer w/ Embeddings	Traces and Spans

Overview

Q&A with Retrieval at a Glance

LLM Input: User Query + retrieved document

LLM Output: Response based on query + document

Evaluation Metrics:

1. Did the LLM answer the question correctly (correctness)

2. For each retrieved document, is the document relevant to answer the user query?

Possibly the most common use-case for creating a LLM application is to connect an LLM to proprietary data such as enterprise documents or video transcriptions. Applications such as these often times are built on top of LLM frameworks such as Langchain or llama_index, which have first-class support for vector store retrievers. Vector Stores enable teams to connect their own data to LLMs. A common application is chatbots looking across a company's knowledge base/context to answer specific questions.

How to Evaluate Retrieval Systems

There are varying degrees of how we can evaluate retrieval systems.

Step 1: First we care if the chatbot is correctly answering the user's questions. Are there certain types of questions the chatbot gets wrong more often?

Step 2: Once we know there's an issue, then we need metrics to trace where specifically did it go wrong. Is the issue with retrieval? Are the documents that the system retrieves irrelevant?

Step 3: If retrieval is not the issue, we should check if we even have the right documents to answer the question.

Question	Metric	Pros	Cons
Is this a bad response to the answer?	User feedback or LLM Eval for Q&A	Most relevant way to measure application	Hard to trace down specifically what to fix
Is the retrieved context relevant?	LLM Eval for Relevance	Directly measures effectiveness of retrieval	Requires additional LLMs calls
Is the knowledge base missing areas of user queries?	Query density (drift) - Phoenix generated	Highlights groups of queries with large distance from context	Identifies broad topics missing from knowledge base, but not small gaps

Using Phoenix Traces & Spans

Visualize the chain of the traces and spans for a Q&A chatbot use case. You can click into specific spans.

When clicking into the retrieval span, you can see the relevance score for each document. This can surface irrelevant context.

Using Phoenix Inferences to Analyze RAG (Retrieval Augmented Generation)

Step 1. Identifying Clusters of Bad Responses

Phoenix surfaces up clusters of similar queries that have poor feedback.

Step 2: Irrelevant Documents Being Retrieved

Phoenix can help uncover when irrelevant context is being retrieved using the LLM Evals for Relevance. You can look at a cluster's aggregate relevance metric with precision @k, NDCG, MRR, etc to identify where to improve. You can also look at a single prompt/response pair and see the relevance of documents.

Step 3: Don't Have Any Documents Close Enough

Phoenix can help you identify if there is context that is missing from your knowledge base. By visualizing query density, you can understand what topics you need to add additional documentation for in order to improve your chatbots responses.

By setting the "primary" dataset as the user queries, and the "corpus" dataset as the context I have in my vector store, I can see if there are clusters of user query embeddings that have no nearby context embeddings, as seen in the example below.

Troubleshooting Tip:

Found a problematic cluster you want to dig into, but don't want to manually sift through all of the prompts and responses? Ask chatGPT to help you understand the make up of the cluster. Try out the colab here

How to Log to Phoenix (as Inferences)

Step 1: Logging Knowledge Base

The first thing we need is to collect some sample from your vector store, to be able to compare against later. This is to able to see if some sections are not being retrieved, or some sections are getting a lot of traffic where you might want to beef up your context or documents in that area.

For more details, visit this page.

id	text	embedding
1	Voyager 2 is a spacecraft used by NASA to expl...	[-0.02785328, -0.04709944, 0.042922903, 0.0559...

corpus_schema = px.Schema(
    id_column_name="id",
    document_column_names=EmbeddingColumnNames(
        vector_column_name="embedding",
        raw_data_column_name="text",
    ),
)

Step 2: Logging Retrieval and Response

We also will be logging the prompt/response pairs from the deployed application.

For more details, visit this page.

query	embedding	retrieved_document_ids	relevance_scores	response
who was the first person that walked on the moon	[-0.0126, 0.0039, 0.0217, ...	[7395, 567965, 323794, ...	[11.30, 7.67, 5.85, ...	Neil Armstrong

primary_schema = Schema(
    prediction_id_column_name="id",
    prompt_column_names=RetrievalEmbeddingColumnNames(
        vector_column_name="embedding",
        raw_data_column_name="query",
        context_retrieval_ids_column_name="retrieved_document_ids",
        context_retrieval_scores_column_name="relevance_scores",
    )
    response_column_names="response",
)