Arize has first-class support for instrumenting OpenAI calls and seeing both input and output messages. We support role types such as system, user, and assistant messages, as well as function calling.
We follow a standardized format for how a trace data should be structured using openinference, which is our open source package based on OpenTelemetry.
Use our code block below to get started using our OpenAIInstrumentor.
# Import open-telemetry dependenciesfrom arize_otel import register_otel, Endpoints# Setup OTEL via our convenience functionregister_otel( endpoints = Endpoints.ARIZE, space_id ="your-space-id", # in app space settings page api_key ="your-api-key", # in app space settings page project_name ="your-project-name", # name this to whatever you would like)# Import the automatic instrumentor from OpenInferencefrom openinference.instrumentation.openai import OpenAIInstrumentor# Finish automatic instrumentationOpenAIInstrumentor().instrument()
Now start asking questions to your LLM app and watch the traces being collected by Arize. For more examples of instrumenting OpenAI applications, check our openinferenece-instrumentation-openai examples.
The OpenAI auto-instrumentation package can be installed via npm.
The example below utilizes the OpenInference JavaScript OpenAI example.
Navigate to the backend folder.
In addition to the above package, sending traces to Arize requires the following packages: @opentelemetry/exporter-trace-otlp-grpc and @grpc/grpc-js. These package can be installed via npm by running the following command in your shell.
instrumentation.ts should be implemented as below (you'll need to install all of the packages imported below in the same manner as above):
/*instrumentation.ts */import { registerInstrumentations } from"@opentelemetry/instrumentation";import { OpenAIInstrumentation } from"@arizeai/openinference-instrumentation-openai";import { ConsoleSpanExporter } from"@opentelemetry/sdk-trace-base";import { NodeTracerProvider, SimpleSpanProcessor,} from"@opentelemetry/sdk-trace-node";import { Resource } from"@opentelemetry/resources";import { OTLPTraceExporter as GrpcOTLPTraceExporter } from"@opentelemetry/exporter-trace-otlp-grpc"; // Arize specificimport { diag, DiagConsoleLogger, DiagLogLevel } from"@opentelemetry/api";import { Metadata } from"@grpc/grpc-js"// For troubleshooting, set the log level to DiagLogLevel.DEBUGdiag.setLogger(newDiagConsoleLogger(),DiagLogLevel.DEBUG);// Arize specific - Create metadata and add your headersconstmetadata=newMetadata();// Your Arize Space and API Keys, which can be found in the UImetadata.set('space_id','your-space-id');metadata.set('api_key','your-api-key');constprovider=newNodeTracerProvider({ resource:newResource({// Arize specific - The name of a new or preexisting model you // want to export spans to"model_id":"your-model-id","model_version":"your-model-version" }),});provider.addSpanProcessor(newSimpleSpanProcessor(newConsoleSpanExporter()));provider.addSpanProcessor(newSimpleSpanProcessor(newGrpcOTLPTraceExporter({ url:"https://otlp.arize.com/v1", metadata, }), ),);registerInstrumentations({ instrumentations: [newOpenAIInstrumentation({})],});provider.register();
If you simultaneously want to send spans to a Phoenix collector, you should also add the following code blocks, from the original instrumentation.ts files.
import { OTLPTraceExporter as ProtoOTLPTraceExporter } from"@opentelemetry/exporter-trace-otlp-proto";// add as another SpanProcessor below the previous SpanProcessorprovider.addSpanProcessor(newSimpleSpanProcessor(newProtoOTLPTraceExporter({// This is the url where your phoenix server is running url:"http://localhost:6006/v1/traces", }), ),);
Follow the steps from the backend and frontend readme. Or simply run:
dockercomposeup--build
to build run the frontend, backend, and Phoenix all at the same time. Navigate to localhost:3000 to begin sending messages to the chatbot and check out your traces in Arize at app.arize.com or Phoenix at localhost:6006.
Tracing Function Calls
Arize simplifies debugging by automatically tracing function calls made by LLMs. With the above code, Arize will receive all function calls and allow users to review the function outputs, arguments, and any parameters used. These traces can be loaded into the prompt playground, where you can experiment with parameters, adjust function definitions, and optimize workflows.