Set up Tracing

We recommend starting with the auto-instrumentation first. If you require more control or want to further customize, you can leverage our OTEL compliant instrumentation API directly.

1. Use Tracing Integrations (Auto-Instrumentation)

Arize makes it easy to log traces with minimal changes by using our tracing integrations. These can then be further customized.

LLM Models	Orchestration Frameworks	Other
OpenAI	LlamaIndex	DSPY
Vertex AI (Gemini)	Langchain / LangGraph	Guardrails AI
AWS Bedrock	Haystack	Prompt Flow
Mistral AI	LiteLLM
Anthropic	CrewAI
	Groq
	Autogen

2. Manual Instrumentation

In some cases, you might want full control over what is being traced in your application. Arize supports OpenTelemetry (OTEL) which means that you can create and customize your spans using the OpenTelemetry Trace API.

Notebooks

LlamaIndex Tracing	Colab Link Colab Link (Arize and Phoenix)
Langchain Tracing	Colab Link
Logging data manually	Colab Link

Example code

Python

Step 1: Install Prerequisities

• Python 3.6 or higher

• OpenTelemetry API and SDK1

• OpenInference Semantic Conventions

pip install opentelemetry-api
pip install opentelemetry-sdk
pip install openinference-semantic-conventions

Step 2: Configuring a Tracer

Manually configuring an OTEL tracer involves some boilerplate code that Arize takes care of for you. We recommend using the register_otel helper method below to configure a tracer.

If you need more control over your tracer's configuration, see the Manually Configuring an OTEL Tracer section below.

import openai
import opentelemetry
import pandas as pd
from openai import OpenAI
from openinference.instrumentation.openai import OpenAIInstrumentor

# Import open-telemetry dependencies
from arize_otel import register_otel, Endpoints

# Setup OTEL via our convenience function.
register_otel(
    endpoints = Endpoints.ARIZE,
    space_id = "your-space-id", # in app space settings page
    api_key = "your-api-key", # in app space settings page
    model_id = "your-model-id", # name this to whatever you would like
)

Step 3: Creating Spans

If you're using a Tracing Integration, these will take care of automatically creating the spans for your application. You can then further customize those spans - for more information check out setting up hybrid instrumentation.

# Because we are using Open AI, we will use this along with our custom instrumentation
OpenAIInstrumentor().instrument()

If you want fully customize the spans, you can do that using manual instrumentation. You'll typically want it to be started as the current span.

def do_work():
    with tracer.start_as_current_span("span-name") as span:
        # do some work that 'span' will track
        print("doing some work...")
        # When the 'with' block goes out of scope, 'span' is closed for you

You can also use start_span to create a span without making it the current span. This is usually done to track concurrent or asynchronous operations.

Creating nested spans

If you have a distinct sub-operation you'd like to track as a part of another one, you can create span to represent the relationship:\

def do_work():
    with tracer.start_as_current_span("parent") as parent:
        # do some work that 'parent' tracks
        print("doing some work...")
        # Create a nested span to track nested work
        with tracer.start_as_current_span("child") as child:
            # do some work that 'child' tracks
            print("doing some nested work...")
            # the nested span is closed when it's out of scope

        # This span is also closed when it goes out of scope

When you view spans in a trace visualization tool, child will be tracked as a nested span under parent.

Creating spans with a decorator

It's common to have a single span track the execution of an entire function. In that scenario, there is a decorator you can use to reduce code:

@tracer.start_as_current_span("do_work")
def do_work():
    print("doing some work...")

Use of the decorator is equivalent to creating the span inside do_work() and ending it when do_work() is finished.

To use the decorator, you must have a tracer instance in scope for your function declaration.

If you need to add attributes or events then it's less convenient to use a decorator.

JS/TS

The code below shows how to setup tracing to export spans to Arize. To see how to use our auto instrumentations see Tracing Integrations (Auto).

Step 1: Install dependencies

npm install @opentelemetry/exporter-trace-otlp-grpc @grpc/grpc-js @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node @opentelemetry/reources @opentelemetry/api

Step 2: Create an instrumentation.ts file

import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { ConsoleSpanExporter } from "@opentelemetry/sdk-trace-base";
import {
  NodeTracerProvider,
  SimpleSpanProcessor,
} from "@opentelemetry/sdk-trace-node";
import { Resource } from "@opentelemetry/resources";
import { 
  OTLPTraceExporter as GrpcOTLPTraceExporter 
} from "@opentelemetry/exporter-trace-otlp-grpc"; // Arize specific
import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";
import { Metadata } from "@grpc/grpc-js"

// For troubleshooting, set the log level to DiagLogLevel.DEBUG
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);

// Arize specific - Create metadata and add your headers
const metadata = new Metadata();

// Your Arize Space and API Keys, which can be found in the UI, see below
metadata.set('space_id', 'your-space-id');
metadata.set('api_key', 'your-api-key');

const provider = new NodeTracerProvider({
  resource: new Resource({
    // Arize specific - The name of a new or preexisting model you 
    // want to export spans to
    "model_id": "your-model-id",
    "model_version": "your-model-version"
  }),
});

provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));
provider.addSpanProcessor(
  new SimpleSpanProcessor(
    new GrpcOTLPTraceExporter({
      url: "https://otlp.arize.com/v1",
      metadata,
    }),
  ),
);

// Add auto instrumentatons here

provider.register();

Step 3: Make sure this file runs at server startup

This file needs to run at the entry point of your application, before any other code on your server runs. You can do this by importing it at the top of your entry point for example in something like server/app.ts:

import "./instrumentation"
// server startup

Or you can import it via a node command when you run your server as outline in the OpenTelemetry documentation using import or require:

# require
npx ts-node --require ./instrumentation.ts app.ts
# import
npx ts-node --import ./instrumentation.ts app.ts
# Or without ts-node something like
node --import ./dist/instrumentation.cjs ./dist/index.cjs

Step 4: Acquiring a tracer Anywhere in your application where you write manual tracing code should call getTracer to acquire a tracer. For example:

import { trace } from '@opentelemetry/api';
//...

const tracer = trace.getTracer(
  'instrumentation-scope-name',
  'instrumentation-scope-version',
);

// You can now use a 'tracer' to create spans!

The values of instrumentation-scope-name and instrumentation-scope-version should uniquely identify the Instrumentation Scope, such as the package, module or class name. While the name is required, the version is still recommended despite being optional.

It’s generally recommended to call getTracer in your app when you need it rather than exporting the tracer instance to the rest of your app. This helps avoid trickier application load issues when other required dependencies are involved.

In the case of the example app, there are two places where a tracer may be acquired with an appropriate Instrumentation Scope:

First, in the application file app.ts (or app.js):

/*app.ts*/
import { trace } from '@opentelemetry/api';
import express, { Express } from 'express';
import { OpenAI } from "openai";

const tracer = trace.getTracer('llm-server', '0.1.0');

const PORT: number = parseInt(process.env.PORT || '8080');
const app: Express = express();

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

app.get('/chat', (req, res) => {
  const message = req.query.message
  let chatCompletion = await openai.chat.completions.create({
    messages: [{ role: "user", content: message }],
    model: "gpt-3.5-turbo",
  });
  res.send(chatCompletion.choices[0].message);
});

app.listen(PORT, () => {
  console.log(`Listening for requests on http://localhost:${PORT}`);
});

Step 5: Create spans Now that you have tracers initialized, you can create spans.

The API of OpenTelemetry JavaScript exposes two methods that allow you to create spans:

• tracer.startSpan: Starts a new span without setting it on context.

• tracer.startActiveSpan: Starts a new span and calls the given callback function passing it the created span as the first argument. The new span gets set in context and this context is activated for the duration of the function call.

In most cases you want to use the latter (tracer.startActiveSpan), as it takes care of setting the span and its context active.

The code below illustrates how to create an active span.

import { trace, Span } from "@opentelemetry/api";
import { SpanKind } from "@opentelemetry/api";
import {
    SemanticConventions,
    OpenInferenceSpanKind,
} from "@arizeai/openinference-semantic-conventions";

/** ... */

export function chat(message: string) {
    const tracer = trace.getTracer(
      'instrumentation-scope-name',
      'instrumentation-scope-version',
    );
    // Create a span. A span must be closed.
    return tracer.startActiveSpan(
        "chat",
        (span: Span) => {
            span.setAttributes({
                [SemanticConventions.OPENINFERENCE_SPAN_KIND]: OpenInferenceSpanKind.chain,
                [SemanticConventions.INPUT_VALUE]: message,
            });
            let chatCompletion = await openai.chat.completions.create({
                messages: [{ role: "user", content: message }],
                model: "gpt-3.5-turbo",
            });
            span.setAttributes({
                attributes: {
                    [SemanticConventions.OUTPUT_VALUE]: chatCompletion.choices[0].message,
                },
            });
            // Be sure to end the span!
            span.end();
            return result;
        }
    );
}

The above instrumented code can now be pasted in the /chat handler. You should now be able to see spans emitted from your app.

Start your app as follows, and then send it requests by visiting http://localhost:8080/chat?message="how long is a pencil" with your browser or curl.

ts-node --require ./instrumentation.ts app.ts

After a while, you should see the spans printed in the console by the ConsoleSpanExporter, something like this:

{
  "traceId": "6cc927a05e7f573e63f806a2e9bb7da8",
  "parentId": undefined,
  "name": "chat",
  "id": "117d98e8add5dc80",
  "kind": 0,
  "timestamp": 1688386291908349,
  "duration": 501,
  "attributes": {
    "openinference.span.kind": "chain"
    "input.value": "how long is a pencil"
  },
  "status": { "code": 0 },
  "events": [],
  "links": []
}

Java

Step 1: Dependency management The OpenTelemetry API provides a set of interfaces for collecting telemetry, but the data is dropped without an implementation. The OpenTelemetry SDK is the implementation of the OpenTelemetry API provided by OpenTelemetry. To use it if you instrument a Java app, begin by installing dependencies.

Here is an example Maven-based pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.arize.app</groupId>
  <artifactId>java-tracing-test</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>java-tracing-test</name>
  <url>http://maven.apache.org</url>
  <dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-bom</artifactId>
            <version>1.38.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
      </dependencies>
  </dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-api</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-sdk</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-logging</artifactId>
    </dependency>
    <dependency>
      <!-- Not managed by opentelemetry-bom -->
      <groupId>io.opentelemetry.semconv</groupId>
      <artifactId>opentelemetry-semconv</artifactId>
      <version>1.25.0-alpha</version>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-otlp</artifactId>
      <version>1.38.0</version>
    </dependency>
  </dependencies>
</project>

Step 2: Initializing and Setting Up the Tracer The first step is to set up the OpenTelemetry SDK with the right headers, resource attributes and endpoint. For example:

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.SpanProcessor;
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
import io.opentelemetry.sdk.trace.export.SpanExporter;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;

public static OpenTelemetrySdk initOpenTelemetry() {
         // Make sure SPACE and API Keys are passed in as env variables for headers
         System.out.println(System.getenv("OTEL_EXPORTER_OTLP_TRACES_HEADERS"));

         // Set up the OTLP exporter with the endpoint and additional headers
         SpanExporter spanExporter = OtlpGrpcSpanExporter.builder()
                 .setEndpoint("https://otlp.arize.com/v1")
                 .build();

         // Create a simple span processor or use a batch span processor for production
         SpanProcessor spanProcessor = SimpleSpanProcessor.create(spanExporter);

         // Set up resource attributes
         Resource resource =
                 Resource.getDefault()
                         .merge(Resource.builder().put("model_id", "java-examples").build());

        //  Set the tracer provider
         SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
                 .addSpanProcessor(spanProcessor)
                 .setResource(resource)
                 .build();

         // Set the OpenTelemetry instance with the tracer provider
         return OpenTelemetrySdk.builder().setTracerProvider(tracerProvider).build();

     }

Now, you can simply call the initOpenTelemetry() function in your Java app in order to initialize OTEL tracing.

OpenTelemetrySdk otelSdk = initOpenTelemetry();

Step 3: Create spans Now that you have tracers initialized, you can create spans.

To create a span, you only need to specify the name of the span and use the method below (make sure to call end() to end the span when you want it to end):

• tracer.spanBuilder.startSpan

The code below illustrates how to create an active span.

// Initialize OTEL Tracer
Tracer tracer = otelSdk.getTracer("my-application", "1.0.0");
                
// Define your span
Span singleAttrSpan = tracer.spanBuilder("span-name").startSpan()

Shutdown thread and tracer Finally, since Opentelemetry's SDK processes spans asynchronously in a separate thread, you need to make sure anything that has been enqueued is processed before exiting the main thread. Thus, it's recommended to make sure you shutdown the application gracefully by adding a shutdown hook at the end. An example of this is below:

finally {
        // Shutdown the thread and tracer
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
                otelSdk.getSdkTracerProvider().shutdown().join(10, java.util.concurrent.TimeUnit.SECONDS);
        }));

This documentation includes material adapted from the OpenTelemetry JS "Instrumentation" documentation, originally authored by the OpenTelemetry Authors, available here. It is used under the Create Commons Attribution 4.0 license (CC BY 4.0). It has been modified here to provide relevant examples for Arize.