Ask or search…
K
Links

LLM Traces

Learn how to use, upload, and debug your LLM Application traces in Arize

Traces & Spans Overview

Traces and spans break down your system's components into discrete inputs and outputs to help debug your LLM application.

Get Started

The Arize Phoenix Tracer is a specialized package developed to streamline trace instrumentation in LLM applications. It leverages an open-source trace collector to simplify tracing application calls which is essential to effectively debug and analyze LLM applications.
Integration & Usage

Collect Traces

Enable Phoenix tracing within LlamaIndex by setting arize_phoenix as the global handler. This will mount Phoenix's OpenInferenceTraceCallback as the global handler. Phoenix uses OpenInference traces - an open-source standard for capturing and storing LLM application traces that enables LLM applications to seamlessly integrate with LLM observability solutions such as Phoenix.
Traces must conform to OpenInference trace format. View the expected column names in the OpenInference trace spec.
Launch a Phoenix server as a trace collector for any LLM application running locally.
import phoenix as px
session = px.launch_app()
The launch_app command will create a URL to use Phoenix. You can access this url again at any time via the session.
🌍 To view the Phoenix app in your browser, visit https://z8rwookkcle1-496ff2e9c6d22116-6060-colab.googleusercontent.com/
📺 To view the Phoenix app in a notebook, run `px.active_session().view()`
📖 For more information on how to use Phoenix, check out https://docs.arize.com/phoenix
The launch_app command will spit out a URL for you to view the Phoenix UI. You can access this url again at any time via the session. Now that phoenix is up and running, you can now run a LlamaIndex or LangChain application OR just run the OpenAI API and debug your application as the traces stream in.
LlamaIndex 🦙
LangChain 🦜🔗
OpenAI
If you are using llama-index>0.8.36 you will be able to instrument your application with LlamaIndex's one-click observability.
# Phoenix can display in real time the traces automatically
# collected from your LlamaIndex application.
import phoenix as px
# Look for a URL in the output to open the App in a browser.
px.launch_app()
# The App is initially empty, but as you proceed with the steps below,
# traces will appear automatically as your LlamaIndex application runs.
import llama_index
llama_index.set_global_handler("arize_phoenix")
# Run your LlamaIndex application and traces
# will be collected and displayed in Phoenix.
# LlamaIndex application initialization may vary
# depending on your application. Below is a simple example:
service_context = ServiceContext.from_defaults(
llm_predictor=LLMPredictor(llm=ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)),
embed_model=OpenAIEmbedding(model="text-embedding-ada-002"),
)
index = load_index_from_storage(
storage_context,
service_context=service_context,
)
query_engine = index.as_query_engine()
# Execute queries
query_engine.query("What is OpenInference tracing?")
See the integrations guide for the full details as well as support for older versions of LlamaIndex
from phoenix.trace.langchain import OpenInferenceTracer, LangChainInstrumentor
# If no exporter is specified, the tracer will export to the locally running Phoenix server
tracer = OpenInferenceTracer()
# If no tracer is specified, a tracer is constructed for you
LangChainInstrumentor(tracer).instrument()
# Initialize your LangChain application
# This might vary on your use-case. An example Chain is shown below
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI
from langchain.embeddings import OpenAIEmbeddings
from langchain.retrievers import KNNRetriever
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")
knn_retriever = KNNRetriever(
index=vectors,
texts=texts,
embeddings=OpenAIEmbeddings(),
)
llm = ChatOpenAI(model_name="gpt-3.5-turbo")
chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="map_reduce",
retriever=knn_retriever,
)
# Execute the chain
response = chain.run("What is OpenInference tracing?")
See this Phoenix integration guide for additional details
from phoenix.trace.tracer import Tracer
from phoenix.trace.openai.instrumentor import OpenAIInstrumentor
from phoenix.trace.exporter import HttpExporter
from phoenix.trace.openai import OpenAIInstrumentor
tracer = Tracer(exporter=HttpExporter())
OpenAIInstrumentor(tracer).instrument()
# Define a conversation with a user message
conversation = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello, can you help me with something?"}
]
# Generate a response from the assistant
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=conversation,
)
# Extract and print the assistant's reply
assistant_reply = response['choices'][0]['message']['content']
#The traces will be available in the Phoenix App for the above messsages
Your application traces should stream in to the Phoenix UI as shown below
Phoenix UI

Export Traces

Run the following command to export your traces from Phoenix.
spans_df = px.active_session().get_spans_dataframe()
This command creates a dataframe object that an be directly ingested to Arize.
An example row of the dataframe

Ingesting data

Schema mapping is not required for OpenInference traces. Arize will automatically infer the column names from your dataframe.
Upload Tracing data to Arize with the Python SDK.
(Cloud import support coming soon)
Python SDK
SPACE_KEY = "SPACE_KEY"
API_KEY = "API_KEY"
if SPACE_KEY == "SPACE_KEY" or API_KEY == "API_KEY":
raise ValueError("❌ NEED TO CHANGE SPACE AND/OR API_KEY")
else:
print("✅ Import and Setup Arize Client Done! Now we can start using Arize!")
arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)
model_id = "generative-spans-tutorial-test" #the model name in Arize
model_version = "1.0" #(optional) the model version
response = arize_client.log_spans(
dataframe=spans_df,
model_id=model_id,
model_version=model_version, #optional
)
# If successful, the server will return a status_code of 200
if response.status_code != 200:
print(f"❌ logging failed with response code {response.status_code}, {response.text}")
else:
print(f"✅ You have successfully logged training set to Arize")

Bring Your Own traces

While we recommend using Phoenix to format your application traces, you can use any trace collector to upload traces to Arize. Simply format your data to conform to the OpenInference trace format and follow the above steps for ingestion.

Debugging Traces in Arize

The following flows are available for in-platform debugging:
If you are interested in Evals support, reach out to [email protected] to get early access