Quickstart: LLM

Learn how to trace your LLM application and run evaluations in Arize

To trace your LLM app and start troubleshooting your LLM calls, you'll need to do the following:

You can also dive right into examples below.

Python:
JS/TS: Github

Install our tracing packages

Run the following commands below to install our open source tracing packages, which works on top of OpenTelemetry. This example below uses openai, and we support many LLM providers (see full list).

Using pip

pip install arize-otel openai openinference-instrumentation-openai opentelemetry-sdk opentelemetry-exporter-otlp

Using conda

conda install -c conda-forge openai openinference-instrumentation-openai opentelemetry-sdk opentelemetry-exporter-otlp

Install with npm

npm install @arizeai/openinference-instrumentation-openai @opentelemetry/exporter-trace-otlp-grpc @grpc/grpc-js

Get your API keys

Go to your space settings in the left navigation, and you will see your API keys on the right hand side. You'll need the space id and API keys for the next part.

Add our tracing code

Arize is an OpenTelemetry collector, which means you can configure your tracer and span processor. For more OTEL configurability, see how to set your tracer for auto instrumentors.

Python and JS/TS examples are shown below

Are you coding with Javascript instead of Python? See our detailed guide on auto-instrumentation or manual instrumentation with Javascript examples.

The following code snippet showcases how to automatically instrument your OpenAI application.

# Import open-telemetry dependencies
from arize_otel import register_otel, Endpoints

# Setup OTEL via our convenience function
register_otel(
    endpoints = Endpoints.ARIZE,
    space_id = "your-space-id", # in app space settings page
    api_key = "your-api-key", # in app space settings page
    model_id = "your-model-id", # name this to whatever you would like
)
# Import the automatic instrumentor from OpenInference
from openinference.instrumentation.openai import OpenAIInstrumentor

# Finish automatic instrumentation
OpenAIInstrumentor().instrument()

Set OpenAI Key:

import os
from getpass import getpass
os.environ["OPENAI_API_KEY"] = getpass("OpenAI API key")

To test, let's send a chat request to OpenAI:

import openai

client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Write a haiku."}],
    max_tokens=20,
)
print(response.choices[0].message.content)

Now start asking questions to your LLM app and watch the traces being collected by Arize. You can also follow our colab guide here.

Google Colab

The following code snippet implements instrumentation for an OpenAI client in typescript.

/*instrumentation.ts */

import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { 
  OpenAIInstrumentation
} from "@arizeai/openinference-instrumentation-openai";
import { ConsoleSpanExporter } from "@opentelemetry/sdk-trace-base";
import {
  NodeTracerProvider,
  BatchSpanProcessor,
} from "@opentelemetry/sdk-trace-node";
import { Resource } from "@opentelemetry/resources";
import { 
  OTLPTraceExporter as GrpcOTLPTraceExporter 
} from "@opentelemetry/exporter-trace-otlp-grpc"; // Arize specific
import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";
import { Metadata } from "@grpc/grpc-js"

// For troubleshooting, set the log level to DiagLogLevel.DEBUG
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);

// Arize specific - Create metadata and add your headers
const metadata = new Metadata();

// Your Arize Space and API Keys, which can be found in the UI
metadata.set('space_id', 'your-space-id');
metadata.set('api_key', 'your-api-key');

const provider = new NodeTracerProvider({
  resource: new Resource({
    // Arize specific - The name of a new or preexisting model you 
    // want to export spans to
    "model_id": "your-model-id",
    "model_version": "your-model-version"
  }),
});

provider.addSpanProcessor(new BatchSpanProcessor(new ConsoleSpanExporter()));
provider.addSpanProcessor(
  new BatchSpanProcessor(
    new GrpcOTLPTraceExporter({
      url: "https://otlp.arize.com/v1",
      metadata,
    }),
  ),
);

registerInstrumentations({
  instrumentations: [new OpenAIInstrumentation({})],
});

provider.register();

You can also follow our example application at the OpenInference github.

openinference/js/examples/openai at main · Arize-ai/openinferenceGitHub

The following code snippet showcases how to automatically instrument your LLM application.

import os
import arize-otel

# Import open-telemetry dependencies
from arize_otel import register_otel, Endpoints

# Setup OTEL via our convenience function
register_otel(
    endpoints = Endpoints.ARIZE,
    space_id = "your-space-id", # in app Space settings page
    api_key = "your-api-key", # in app Space settings page
    model_id = "your-model-id",
)
# Import the automatic instrumentor from OpenInference
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor

# Finish automatic instrumentation
LlamaIndexInstrumentor().instrument()

To test, you can create a simple RAG application using LlamaIndex.

from gcsfs import GCSFileSystem
from llama_index.core import (
    Settings,
    StorageContext,
    load_index_from_storage,
)
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

file_system = GCSFileSystem(project="public-assets-275721")
index_path = "arize-phoenix-assets/datasets/unstructured/llm/llama-index/arize-docs/index/"
storage_context = StorageContext.from_defaults(
    fs=file_system,
    persist_dir=index_path,
)

Settings.llm = OpenAI(model="gpt-4-turbo-preview")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-ada-002")
index = load_index_from_storage(
    storage_context,
)
query_engine = index.as_query_engine()

response = query_engine.query("What is Arize and how can it help me as an AI Engineer?")

Now start asking questions to your LLM app and watch the traces being collected by Arize.

The following code snippet showcases how to automatically instrument your LLM application.

import os
import arize-otel

# Import open-telemetry dependencies
from arize_otel import register_otel, Endpoints

# Setup OTEL via our convenience function
register_otel(
    endpoints = Endpoints.ARIZE,
    space_id = "your-space-id", # in app Space settings page
    api_key = "your-api-key", # in app Space settings page
    model_id = "your-model-id",
)
# Import the automatic instrumentor from OpenInference
from openinference.instrumentation.langchain import LangChainInstrumentor

# Finish automatic instrumentation
LangChainInstrumentor().instrument()

To test, you can create a simple RAG application using Langchain.

from langchain.chains import RetrievalQA
from langchain.retrievers import KNNRetriever
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

knn_retriever = KNNRetriever(
    index=np.stack(df["text_vector"]),
    texts=df["text"].tolist(),
    embeddings=OpenAIEmbeddings(),
)
chain_type = "stuff"  # stuff, refine, map_reduce, and map_rerank
chat_model_name = "gpt-3.5-turbo"
llm = ChatOpenAI(model_name=chat_model_name)
chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type=chain_type,
    retriever=knn_retriever,
    metadata={"application_type": "question_answering"},
)

response = chain.invoke("What is Arize and how can it help me as an AI Engineer?")

Now start asking questions to your LLM app and watch the traces being collected by Arize.

Run your LLM application

Once you've executed a sufficient number of queries (or chats) to your application, you can view the details on the LLM Tracing page.

To continue with this guide, go to the quickstart: evaluation to add evaluation labels to your traces!

Next steps

Dive deeper into the following topics to keep improving your LLM application!

Tracing

Evaluation

Prompt engineering

Production monitoring

Retrieval

PreviousExamples NextConcepts: Tracing

Last updated 17 days ago