LogoLogo
Python SDKSlack
  • Documentation
  • Cookbooks
  • Self-Hosting
  • Release Notes
  • Reference
  • Arize AI
  • Quickstarts
  • โœจArize Copilot
  • Arize AI for Agents
  • Concepts
    • Agent Evaluation
    • Tracing
      • What is OpenTelemetry?
      • What is OpenInference?
      • Openinference Semantic Conventions
    • Evaluation
  • ๐ŸงชDevelop
    • Quickstart: Experiments
    • Datasets
      • Create a dataset
      • Update a dataset
      • Export a dataset
    • Experiments
      • Run experiments
      • Run experiments with code
        • Experiments SDK differences in AX vs Phoenix
        • Log experiment results via SDK
      • Evaluate experiments
      • Evaluate experiment with code
      • CI/CD with experiments
        • Github Action Basics
        • Gitlab CI/CD Basics
      • Download experiment
    • Prompt Playground
      • Use tool calling
      • Use image inputs
      • Replay spans
      • Compare prompts side-by-side
      • Load a dataset into playground
      • Save playground outputs as an experiment
      • โœจCopilot: prompt builder
    • Playground Integrations
      • OpenAI
      • Azure OpenAI
      • AWS Bedrock
      • VertexAI
      • Custom LLM Models
    • Prompt Hub
  • ๐Ÿง Evaluate
    • Online Evals
      • Run evaluations in the UI
      • Run evaluations with code
      • Test LLM evaluator in playground
      • View task details & logs
      • โœจCopilot: Eval Builder
      • โœจCopilot: Eval Analysis
      • โœจCopilot: RAG Analysis
    • Experiment Evals
    • LLM as a Judge
      • Custom Eval Templates
      • Arize Templates
        • Agent Tool Calling
        • Agent Tool Selection
        • Agent Parameter Extraction
        • Agent Path Convergence
        • Agent Planning
        • Agent Reflection
        • Hallucinations
        • Q&A on Retrieved Data
        • Summarization
        • Code Generation
        • Toxicity
        • AI vs Human (Groundtruth)
        • Citation
        • User Frustration
        • SQL Generation
    • Code Evaluations
    • Human Annotations
  • ๐Ÿ”ญObserve
    • Quickstart: Tracing
    • Tracing
      • Setup tracing
      • Trace manually
        • Trace inputs and outputs
        • Trace function calls
        • Trace LLM, Retriever and Tool Spans
        • Trace prompt templates & variables
        • Trace as Inferences
        • Send Traces from Phoenix -> Arize
        • Advanced Tracing (OTEL) Examples
      • Add metadata
        • Add events, exceptions and status
        • Logging Latent Metadata
        • Add attributes, metadata and tags
        • Send data to a specific project
        • Get the current span context and tracer
      • Configure tracing options
        • Configure OTEL tracer
        • Mask span attributes
        • Redact sensitive data from traces
        • Instrument with OpenInference helpers
      • Query traces
        • Filter Traces
          • Time Filtering
        • Export Traces
        • โœจAI Powered Search & Filter
        • โœจAI Powered Trace Analysis
        • โœจAI Span Analysis & Evaluation
    • Tracing Integrations
      • OpenAI
      • OpenAI Agents SDK
      • LlamaIndex
      • LlamaIndex Workflows
      • LangChain
      • LangGraph
      • Hugging Face smolagents
      • Autogen
      • Google GenAI (Gemini)
      • Model Context Protocol (MCP)
      • Vertex AI
      • Amazon Bedrock
      • Amazon Bedrock Agents
      • MistralAI
      • Anthropic
      • LangFlow
      • Haystack
      • LiteLLM
      • CrewAI
      • Groq
      • DSPy
      • Guardrails AI
      • Prompt flow
      • Vercel AI SDK
      • Llama
      • Together AI
      • OpenTelemetry (arize-otel)
      • BeeAI
    • Evals on Traces
    • Guardrails
    • Sessions
    • Dashboards
      • Dashboard Widgets
      • Tracking Token Usage
      • โœจCopilot: Dashboard Widget Creation
    • Monitors
      • Integrations: Monitors
        • Slack
          • Manual Setup
        • OpsGenie
        • PagerDuty
      • LLM Red Teaming
    • Custom Metrics & Analytics
      • Arize Query Language Syntax
        • Conditionals and Filters
        • All Operators
        • All Functions
      • Custom Metric Examples
      • โœจCopilot: ArizeQL Generator
  • ๐Ÿ“ˆMachine Learning
    • Machine Learning
      • User Guide: ML
      • Quickstart: ML
      • Concepts: ML
        • What Is A Model Schema
        • Delayed Actuals and Tags
        • ML Glossary
      • How To: ML
        • Upload Data to Arize
          • Pandas SDK Example
          • Local File Upload
            • File Upload FAQ
          • Table Ingestion Tuning
          • Wildcard Paths for Cloud Storage
          • Troubleshoot Data Upload
          • Sending Data FAQ
        • Monitors
          • ML Monitor Types
          • Configure Monitors
            • Notifications Providers
          • Programmatically Create Monitors
          • Best Practices for Monitors
        • Dashboards
          • Dashboard Widgets
          • Dashboard Templates
            • Model Performance
            • Pre-Production Performance
            • Feature Analysis
            • Drift
          • Programmatically Create Dashboards
        • Performance Tracing
          • Time Filtering
          • โœจCopilot: Performance Insights
        • Drift Tracing
          • โœจCopilot: Drift Insights
          • Data Distribution Visualization
          • Embeddings for Tabular Data (Multivariate Drift)
        • Custom Metrics
          • Arize Query Language Syntax
            • Conditionals and Filters
            • All Operators
            • All Functions
          • Custom Metric Examples
          • Custom Metrics Query Language
          • โœจCopilot: ArizeQL Generator
        • Troubleshoot Data Quality
          • โœจCopilot: Data Quality Insights
        • Explainability
          • Interpreting & Analyzing Feature Importance Values
          • SHAP
          • Surrogate Model
          • Explainability FAQ
          • Model Explainability
        • Bias Tracing (Fairness)
        • Export Data to Notebook
        • Automate Model Retraining
        • ML FAQ
      • Use Cases: ML
        • Binary Classification
          • Fraud
          • Insurance
        • Multi-Class Classification
        • Regression
          • Lending
          • Customer Lifetime Value
          • Click-Through Rate
        • Timeseries Forecasting
          • Demand Forecasting
          • Churn Forecasting
        • Ranking
          • Collaborative Filtering
          • Search Ranking
        • Natural Language Processing (NLP)
        • Common Industry Use Cases
      • Integrations: ML
        • Google BigQuery
          • GBQ Views
          • Google BigQuery FAQ
        • Snowflake
          • Snowflake Permissions Configuration
        • Databricks
        • Google Cloud Storage (GCS)
        • Azure Blob Storage
        • AWS S3
          • Private Image Link Access Via AWS S3
        • Kafka
        • Airflow Retrain
        • Amazon EventBridge Retrain
        • MLOps Partners
          • Algorithmia
          • Anyscale
          • Azure & Databricks
          • BentoML
          • CML (DVC)
          • Deepnote
          • Feast
          • Google Cloud ML
          • Hugging Face
          • LangChain ๐Ÿฆœ๐Ÿ”—
          • MLflow
          • Neptune
          • Paperspace
          • PySpark
          • Ray Serve (Anyscale)
          • SageMaker
            • Batch
            • RealTime
            • Notebook Instance with Greater than 20GB of Data
          • Spell
          • UbiOps
          • Weights & Biases
      • API Reference: ML
        • Python SDK
          • Pandas Batch Logging
            • Client
            • log
            • Schema
            • TypedColumns
            • EmbeddingColumnNames
            • ObjectDetectionColumnNames
            • PromptTemplateColumnNames
            • LLMConfigColumnNames
            • LLMRunMetadataColumnNames
            • NLP_Metrics
            • AutoEmbeddings
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
          • Single Record Logging
            • Client
            • log
            • TypedValue
            • Ranking
            • Multi-Class
            • Object Detection
            • Embedding
            • LLMRunMetadata
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
        • Java SDK
          • Constructor
          • log
          • bulkLog
          • logValidationRecords
          • logTrainingRecords
        • R SDK
          • Client$new()
          • Client$log()
        • Rest API
    • Computer Vision
      • How to: CV
        • Generate Embeddings
          • How to Generate Your Own Embedding
          • Let Arize Generate Your Embeddings
        • Embedding & Cluster Analyzer
        • โœจCopilot: Embedding Summarization
        • Similarity Search
        • Embedding Drift
        • Embeddings FAQ
      • Integrations: CV
      • Use Cases: CV
        • Image Classification
        • Image Segmentation
        • Object Detection
      • API Reference: CV
Powered by GitBook

Support

  • Chat Us On Slack
  • support@arize.com

Get Started

  • Signup For Free
  • Book A Demo

Copyright ยฉ 2025 Arize AI, Inc

On this page
  • 1. Use Tracing Integrations (Auto-Instrumentation)
  • 2. Manual Instrumentation

Was this helpful?

  1. Observe
  2. Tracing

Setup tracing

Last updated 17 days ago

Was this helpful?

We recommend starting with the auto-instrumentation first. If you require more control or want to further customize, you can leverage our OTEL compliant instrumentation API directly.

1. Use Tracing Integrations (Auto-Instrumentation)

Arize makes it easy to log traces with minimal changes by using our tracing integrations. These can then be further .

2. Manual Instrumentation

In some cases, you might want full control over what is being traced in your application. Arize supports OpenTelemetry (OTEL) which means that you can create and customize your spans using the OpenTelemetry Trace API.

Notebooks

Logging data manually

LlamaIndex m

Langchain Tracing

Example code

Step 1: Install Prerequisities

  • Python 3.6 or higher

  • OpenTelemetry API and SDK1

pip install opentelemetry-api
pip install opentelemetry-sdk
pip install openinference-semantic-conventions

Step 2: Configuring a Tracer

Manually configuring an OTEL tracer involves some boilerplate code that Arize takes care of for you. We recommend using the register helper function below to configure a tracer.

If you need more control over your tracer's configuration, see the section below.

import openai
import opentelemetry
import pandas as pd
from openai import OpenAI
from openinference.instrumentation.openai import OpenAIInstrumentor
from arize.otel import register

# Setup OTel via our convenience function
tracer_provider = register(
    space_id = "your-space-id", # in app space settings page
    api_key = "your-api-key", # in app space settings page
    project_name = "your-project-name", # name this to whatever you would like
)

Step 3: Creating Spans

# Because we are using Open AI, we will use this along with our custom instrumentation
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

If you want fully customize the spans, you can do that using manual instrumentation.

First we should set the tracer for our manual spans below

from opentelemetry import trace

trace.set_tracer_provider(tracer_provider)
tracer = trace.get_tracer(__name__)

Next we create spans by starting spans and defining our name and other attributes

def do_work():
    with tracer.start_as_current_span("span-name") as span:
        # do some work that 'span' will track
        print("doing some work...")
        # When the 'with' block goes out of scope, 'span' is closed for you

You can also use start_span to create a span without making it the current span. This is usually done to track concurrent or asynchronous operations.

Creating nested spans

If you have a distinct sub-operation you'd like to track as a part of another one, you can create span to represent the relationship:\

def do_work():
    with tracer.start_as_current_span("parent") as parent:
        # do some work that 'parent' tracks
        print("doing some work...")
        # Create a nested span to track nested work
        with tracer.start_as_current_span("child") as child:
            # do some work that 'child' tracks
            print("doing some nested work...")
            # the nested span is closed when it's out of scope

        # This span is also closed when it goes out of scope

When you view spans in a trace visualization tool, child will be tracked as a nested span under parent.

Creating spans with a decorator

It's common to have a single span track the execution of an entire function. In that scenario, there is a decorator you can use to reduce code:

@tracer.start_as_current_span("do_work")
def do_work():
    print("doing some work...")

Use of the decorator is equivalent to creating the span inside do_work() and ending it when do_work() is finished.

To use the decorator, you must have a tracer instance in scope for your function declaration.

The code below shows how to setup tracing to export spans to Arize. To see how to use our auto instrumentations see Tracing Integrations.

Step 1: Install dependencies

npm install @opentelemetry/exporter-trace-otlp-grpc @grpc/grpc-js @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node @opentelemetry/resources @opentelemetry/api

Step 2: Create an instrumentation.ts file

import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { ConsoleSpanExporter } from "@opentelemetry/sdk-trace-base";
import {
  NodeTracerProvider,
  SimpleSpanProcessor,
} from "@opentelemetry/sdk-trace-node";
import { Resource } from "@opentelemetry/resources";
import { 
  OTLPTraceExporter as GrpcOTLPTraceExporter 
} from "@opentelemetry/exporter-trace-otlp-grpc"; // Arize specific
import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api";
import { Metadata } from "@grpc/grpc-js"

// For troubleshooting, set the log level to DiagLogLevel.DEBUG
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);

// Arize specific - Create metadata and add your headers
const metadata = new Metadata();

// Your Arize Space and API Keys, which can be found in the UI, see below
metadata.set('space_id', 'your-space-id');
metadata.set('api_key', 'your-api-key');

const provider = new NodeTracerProvider({
  resource: new Resource({
    // Arize specific - The name of a new or preexisting model you 
    // want to export spans to
    "model_id": "your-model-id",
    "model_version": "your-model-version"
  }),
});

provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));
provider.addSpanProcessor(
  new SimpleSpanProcessor(
    new GrpcOTLPTraceExporter({
      url: "https://otlp.arize.com/v1",
      metadata,
    }),
  ),
);

// Add auto instrumentatons here

provider.register();

Step 3: Make sure this file runs at server startup

This file needs to run at the entry point of your application, before any other code on your server runs. You can do this by importing it at the top of your entry point for example in something like server/app.ts:

import "./instrumentation"
// server startup
# require
npx ts-node --require ./instrumentation.ts app.ts
# import
npx ts-node --import ./instrumentation.ts app.ts
# Or without ts-node something like
node --import ./dist/instrumentation.cjs ./dist/index.cjs

Step 4: Acquiring a tracer Anywhere in your application where you write manual tracing code should call getTracer to acquire a tracer. For example:

import { trace } from '@opentelemetry/api';
//...

const tracer = trace.getTracer(
  'instrumentation-scope-name',
  'instrumentation-scope-version',
);

// You can now use a 'tracer' to create spans!

Itโ€™s generally recommended to call getTracer in your app when you need it rather than exporting the tracer instance to the rest of your app. This helps avoid trickier application load issues when other required dependencies are involved.

First, in the application file app.ts (or app.js):

/*app.ts*/
import { trace } from '@opentelemetry/api';
import express, { Express } from 'express';
import { OpenAI } from "openai";

const tracer = trace.getTracer('llm-server', '0.1.0');

const PORT: number = parseInt(process.env.PORT || '8080');
const app: Express = express();

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

app.get('/chat', (req, res) => {
  const message = req.query.message
  let chatCompletion = await openai.chat.completions.create({
    messages: [{ role: "user", content: message }],
    model: "gpt-3.5-turbo",
  });
  res.send(chatCompletion.choices[0].message);
});

app.listen(PORT, () => {
  console.log(`Listening for requests on http://localhost:${PORT}`);
});

The API of OpenTelemetry JavaScript exposes two methods that allow you to create spans:

In most cases you want to use the latter (tracer.startActiveSpan), as it takes care of setting the span and its context active.

The code below illustrates how to create an active span.

import { trace, Span } from "@opentelemetry/api";
import { SpanKind } from "@opentelemetry/api";
import {
    SemanticConventions,
    OpenInferenceSpanKind,
} from "@arizeai/openinference-semantic-conventions";

/** ... */

export function chat(message: string) {
    const tracer = trace.getTracer(
      'instrumentation-scope-name',
      'instrumentation-scope-version',
    );
    // Create a span. A span must be closed.
    return tracer.startActiveSpan(
        "chat",
        (span: Span) => {
            span.setAttributes({
                [SemanticConventions.OPENINFERENCE_SPAN_KIND]: OpenInferenceSpanKind.chain,
                [SemanticConventions.INPUT_VALUE]: message,
            });
            let chatCompletion = await openai.chat.completions.create({
                messages: [{ role: "user", content: message }],
                model: "gpt-3.5-turbo",
            });
            span.setAttributes({
                attributes: {
                    [SemanticConventions.OUTPUT_VALUE]: chatCompletion.choices[0].message,
                },
            });
            // Be sure to end the span!
            span.end();
            return result;
        }
    );
}

The above instrumented code can now be pasted in the /chat handler. You should now be able to see spans emitted from your app.

Start your app as follows, and then send it requests by visiting http://localhost:8080/chat?message="how long is a pencil" with your browser or curl.

ts-node --require ./instrumentation.ts app.ts

After a while, you should see the spans printed in the console by the ConsoleSpanExporter, something like this:

{
  "traceId": "6cc927a05e7f573e63f806a2e9bb7da8",
  "parentId": undefined,
  "name": "chat",
  "id": "117d98e8add5dc80",
  "kind": 0,
  "timestamp": 1688386291908349,
  "duration": 501,
  "attributes": {
    "openinference.span.kind": "chain"
    "input.value": "how long is a pencil"
  },
  "status": { "code": 0 },
  "events": [],
  "links": []
}

Step 1: Dependency management The OpenTelemetry API provides a set of interfaces for collecting telemetry, but the data is dropped without an implementation. The OpenTelemetry SDK is the implementation of the OpenTelemetry API provided by OpenTelemetry. To use it if you instrument a Java app, begin by installing dependencies.

Here is an example Maven-based pom.xml file:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.arize.app</groupId>
  <artifactId>java-tracing-test</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>java-tracing-test</name>
  <url>http://maven.apache.org</url>
  <dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-bom</artifactId>
            <version>1.38.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
      </dependencies>
  </dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-api</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-sdk</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-logging</artifactId>
    </dependency>
    <dependency>
      <!-- Not managed by opentelemetry-bom -->
      <groupId>io.opentelemetry.semconv</groupId>
      <artifactId>opentelemetry-semconv</artifactId>
      <version>1.25.0-alpha</version>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-otlp</artifactId>
      <version>1.38.0</version>
    </dependency>
  </dependencies>
</project>

Step 2: Initializing and Setting Up the Tracer The first step is to set up the OpenTelemetry SDK with the right headers, resource attributes and endpoint. For example:

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.SpanProcessor;
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
import io.opentelemetry.sdk.trace.export.SpanExporter;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;

public static OpenTelemetrySdk initOpenTelemetry() {
         // Make sure SPACE and API Keys are passed in as env variables for headers
         System.out.println(System.getenv("OTEL_EXPORTER_OTLP_TRACES_HEADERS"));

         // Set up the OTLP exporter with the endpoint and additional headers
         SpanExporter spanExporter = OtlpGrpcSpanExporter.builder()
                 .setEndpoint("https://otlp.arize.com/v1")
                 .build();

         // Create a simple span processor or use a batch span processor for production
         SpanProcessor spanProcessor = SimpleSpanProcessor.create(spanExporter);

         // Set up resource attributes
         Resource resource =
                 Resource.getDefault()
                         .merge(Resource.builder().put("model_id", "java-examples").build());

        //  Set the tracer provider
         SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
                 .addSpanProcessor(spanProcessor)
                 .setResource(resource)
                 .build();

         // Set the OpenTelemetry instance with the tracer provider
         return OpenTelemetrySdk.builder().setTracerProvider(tracerProvider).build();

     }

Now, you can simply call the initOpenTelemetry() function in your Java app in order to initialize OTEL tracing.

OpenTelemetrySdk otelSdk = initOpenTelemetry();

To create a span, you only need to specify the name of the span and use the method below (make sure to call end() to end the span when you want it to end):

The code below illustrates how to create an active span.

// Initialize OTEL Tracer
Tracer tracer = otelSdk.getTracer("my-application", "1.0.0");
                
// Define your span
Span singleAttrSpan = tracer.spanBuilder("span-name").startSpan()

Shutdown thread and tracer Finally, since Opentelemetry's SDK processes spans asynchronously in a separate thread, you need to make sure anything that has been enqueued is processed before exiting the main thread. Thus, it's recommended to make sure you shutdown the application gracefully by adding a shutdown hook at the end. An example of this is below:

finally {
        // Shutdown the thread and tracer
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
                otelSdk.getSdkTracerProvider().shutdown().join(10, java.util.concurrent.TimeUnit.SECONDS);
        }));

If you're using a Tracing Integration, these will take care of automatically creating the spans for your application. You can then further customize those spans - for more information check out setting up .

If you need to add or then it's less convenient to use a decorator.

Or you can import it via a node command when you run your server as outline in the using import or require:

The values of instrumentation-scope-name and instrumentation-scope-version should uniquely identify the , such as the package, module or class name. While the name is required, the version is still recommended despite being optional.

In the case of the , there are two places where a tracer may be acquired with an appropriate Instrumentation Scope:

Step 5: Create spans Now that you have initialized, you can create .

: Starts a new span without setting it on context.

: Starts a new span and calls the given callback function passing it the created span as the first argument. The new span gets set in context and this context is activated for the duration of the function call.

Step 3: Create spans Now that you have initialized, you can create .

This documentation includes material adapted from the OpenTelemetry JS "Instrumentation" documentation, originally authored by the OpenTelemetry Authors, available . It is used under the Create Commons Attribution 4.0 license (). It has been modified here to provide relevant examples for Arize.

๐Ÿ”ญ
hybrid instrumentation
attributes
events
OpenTelemetry documentation
Instrumentation Scope
example app
tracers
spans
tracer.startSpan
tracer.startActiveSpan
tracers
spans
tracer.spanBuilder.startSpan
here
CC BY 4.0
Colab Link
Colab Link
Colab Link (Arize and Phoenix)
Colab Link

LLM Model Providers

Agent Frameworks

Other

customized
OpenInference Semantic Conventions
Manually Configuring an OTEL Tracer

Anthropic Claude
AWS Bedrock
Google GenAI (Gemini)
Groq
OpenAI
Mistral AI
Autogen
CrewAI
IBM BeeAI
Huggingface smolagents
LangGraph
LlamaIndex Workflows
OpenAI Agents
DSPy
Guardrails AI
Haystack
LiteLLM
Llama
OpenTelemetry
Promptflow
Together AI
Vercel AI