LogoLogo
Python SDKSlack
  • Documentation
  • Cookbooks
  • Self-Hosting
  • Release Notes
  • Reference
  • Arize AI
  • Quickstarts
  • ✨Arize Copilot
  • Arize AI for Agents
  • Concepts
    • Agent Evaluation
    • Tracing
      • What is OpenTelemetry?
      • What is OpenInference?
      • Openinference Semantic Conventions
    • Evaluation
  • 🧪Develop
    • Quickstart: Experiments
    • Datasets
      • Create a dataset
      • Update a dataset
      • Export a dataset
    • Experiments
      • Run experiments
      • Run experiments with code
        • Experiments SDK differences in AX vs Phoenix
        • Log experiment results via SDK
      • Evaluate experiments
      • Evaluate experiment with code
      • CI/CD with experiments
        • Github Action Basics
        • Gitlab CI/CD Basics
      • Download experiment
    • Prompt Playground
      • Use tool calling
      • Use image inputs
      • Replay spans
      • Compare prompts side-by-side
      • Load a dataset into playground
      • Save playground outputs as an experiment
      • ✨Copilot: prompt builder
    • Playground Integrations
      • OpenAI
      • Azure OpenAI
      • AWS Bedrock
      • VertexAI
      • Custom LLM Models
    • Prompt Hub
  • 🧠Evaluate
    • Online Evals
      • Run evaluations in the UI
      • Run evaluations with code
      • Test LLM evaluator in playground
      • View task details & logs
      • ✨Copilot: Eval Builder
      • ✨Copilot: Eval Analysis
      • ✨Copilot: RAG Analysis
    • Experiment Evals
    • LLM as a Judge
      • Custom Eval Templates
      • Arize Templates
        • Agent Tool Calling
        • Agent Tool Selection
        • Agent Parameter Extraction
        • Agent Path Convergence
        • Agent Planning
        • Agent Reflection
        • Hallucinations
        • Q&A on Retrieved Data
        • Summarization
        • Code Generation
        • Toxicity
        • AI vs Human (Groundtruth)
        • Citation
        • User Frustration
        • SQL Generation
    • Code Evaluations
    • Human Annotations
  • 🔭Observe
    • Quickstart: Tracing
    • Tracing
      • Setup tracing
      • Trace manually
        • Trace inputs and outputs
        • Trace function calls
        • Trace LLM, Retriever and Tool Spans
        • Trace prompt templates & variables
        • Trace as Inferences
        • Send Traces from Phoenix -> Arize
        • Advanced Tracing (OTEL) Examples
      • Add metadata
        • Add events, exceptions and status
        • Logging Latent Metadata
        • Add attributes, metadata and tags
        • Send data to a specific project
        • Get the current span context and tracer
      • Configure tracing options
        • Configure OTEL tracer
        • Mask span attributes
        • Redact sensitive data from traces
        • Instrument with OpenInference helpers
      • Query traces
        • Filter Traces
          • Time Filtering
        • Export Traces
        • ✨AI Powered Search & Filter
        • ✨AI Powered Trace Analysis
        • ✨AI Span Analysis & Evaluation
    • Tracing Integrations
      • OpenAI
      • OpenAI Agents SDK
      • LlamaIndex
      • LlamaIndex Workflows
      • LangChain
      • LangGraph
      • Hugging Face smolagents
      • Autogen
      • Google GenAI (Gemini)
      • Model Context Protocol (MCP)
      • Vertex AI
      • Amazon Bedrock
      • Amazon Bedrock Agents
      • MistralAI
      • Anthropic
      • LangFlow
      • Haystack
      • LiteLLM
      • CrewAI
      • Groq
      • DSPy
      • Guardrails AI
      • Prompt flow
      • Vercel AI SDK
      • Llama
      • Together AI
      • OpenTelemetry (arize-otel)
      • BeeAI
    • Evals on Traces
    • Guardrails
    • Sessions
    • Dashboards
      • Dashboard Widgets
      • Tracking Token Usage
      • ✨Copilot: Dashboard Widget Creation
    • Monitors
      • Integrations: Monitors
        • Slack
          • Manual Setup
        • OpsGenie
        • PagerDuty
      • LLM Red Teaming
    • Custom Metrics & Analytics
      • Arize Query Language Syntax
        • Conditionals and Filters
        • All Operators
        • All Functions
      • Custom Metric Examples
      • ✨Copilot: ArizeQL Generator
  • 📈Machine Learning
    • Machine Learning
      • User Guide: ML
      • Quickstart: ML
      • Concepts: ML
        • What Is A Model Schema
        • Delayed Actuals and Tags
        • ML Glossary
      • How To: ML
        • Upload Data to Arize
          • Pandas SDK Example
          • Local File Upload
            • File Upload FAQ
          • Table Ingestion Tuning
          • Wildcard Paths for Cloud Storage
          • Troubleshoot Data Upload
          • Sending Data FAQ
        • Monitors
          • ML Monitor Types
          • Configure Monitors
            • Notifications Providers
          • Programmatically Create Monitors
          • Best Practices for Monitors
        • Dashboards
          • Dashboard Widgets
          • Dashboard Templates
            • Model Performance
            • Pre-Production Performance
            • Feature Analysis
            • Drift
          • Programmatically Create Dashboards
        • Performance Tracing
          • Time Filtering
          • ✨Copilot: Performance Insights
        • Drift Tracing
          • ✨Copilot: Drift Insights
          • Data Distribution Visualization
          • Embeddings for Tabular Data (Multivariate Drift)
        • Custom Metrics
          • Arize Query Language Syntax
            • Conditionals and Filters
            • All Operators
            • All Functions
          • Custom Metric Examples
          • Custom Metrics Query Language
          • ✨Copilot: ArizeQL Generator
        • Troubleshoot Data Quality
          • ✨Copilot: Data Quality Insights
        • Explainability
          • Interpreting & Analyzing Feature Importance Values
          • SHAP
          • Surrogate Model
          • Explainability FAQ
          • Model Explainability
        • Bias Tracing (Fairness)
        • Export Data to Notebook
        • Automate Model Retraining
        • ML FAQ
      • Use Cases: ML
        • Binary Classification
          • Fraud
          • Insurance
        • Multi-Class Classification
        • Regression
          • Lending
          • Customer Lifetime Value
          • Click-Through Rate
        • Timeseries Forecasting
          • Demand Forecasting
          • Churn Forecasting
        • Ranking
          • Collaborative Filtering
          • Search Ranking
        • Natural Language Processing (NLP)
        • Common Industry Use Cases
      • Integrations: ML
        • Google BigQuery
          • GBQ Views
          • Google BigQuery FAQ
        • Snowflake
          • Snowflake Permissions Configuration
        • Databricks
        • Google Cloud Storage (GCS)
        • Azure Blob Storage
        • AWS S3
          • Private Image Link Access Via AWS S3
        • Kafka
        • Airflow Retrain
        • Amazon EventBridge Retrain
        • MLOps Partners
          • Algorithmia
          • Anyscale
          • Azure & Databricks
          • BentoML
          • CML (DVC)
          • Deepnote
          • Feast
          • Google Cloud ML
          • Hugging Face
          • LangChain 🦜🔗
          • MLflow
          • Neptune
          • Paperspace
          • PySpark
          • Ray Serve (Anyscale)
          • SageMaker
            • Batch
            • RealTime
            • Notebook Instance with Greater than 20GB of Data
          • Spell
          • UbiOps
          • Weights & Biases
      • API Reference: ML
        • Python SDK
          • Pandas Batch Logging
            • Client
            • log
            • Schema
            • TypedColumns
            • EmbeddingColumnNames
            • ObjectDetectionColumnNames
            • PromptTemplateColumnNames
            • LLMConfigColumnNames
            • LLMRunMetadataColumnNames
            • NLP_Metrics
            • AutoEmbeddings
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
          • Single Record Logging
            • Client
            • log
            • TypedValue
            • Ranking
            • Multi-Class
            • Object Detection
            • Embedding
            • LLMRunMetadata
            • utils.types.ModelTypes
            • utils.types.Metrics
            • utils.types.Environments
        • Java SDK
          • Constructor
          • log
          • bulkLog
          • logValidationRecords
          • logTrainingRecords
        • R SDK
          • Client$new()
          • Client$log()
        • Rest API
    • Computer Vision
      • How to: CV
        • Generate Embeddings
          • How to Generate Your Own Embedding
          • Let Arize Generate Your Embeddings
        • Embedding & Cluster Analyzer
        • ✨Copilot: Embedding Summarization
        • Similarity Search
        • Embedding Drift
        • Embeddings FAQ
      • Integrations: CV
      • Use Cases: CV
        • Image Classification
        • Image Segmentation
        • Object Detection
      • API Reference: CV
Powered by GitBook

Support

  • Chat Us On Slack
  • support@arize.com

Get Started

  • Signup For Free
  • Book A Demo

Copyright © 2025 Arize AI, Inc

On this page
  • 1. Manual Context Propagation
  • Propagation with Async Functions
  • Propagation Across Different Microservices
  • Propagation with Concurrent Threads
  • 2. Creating Custom Decorators
  • 3. Filtering Spans Based on Specific Attributes
  • Custom Sampling Basics
  • 4. Inheriting Context Attributes in Manual Spans

Was this helpful?

  1. Observe
  2. Tracing
  3. Trace manually

Advanced Tracing (OTEL) Examples

Manual Context Propagation, Custom Decorators, Custom Exports and Sampling

This documentation provides advanced use cases and examples, including manual context propagation, custom decorators, custom sampling/filtering, and more. These scenarios address real-world needs such as asynchronous execution, multi-service flows and specialized exporters or decorators for observability platforms like Arize.

1. Manual Context Propagation

Context Propagation in OpenTelemetry ensures that the current tracing context (i.e., the currently active span and its metadata) is available whenever you switch threads, tasks, or processes. This is particularly relevant if your code spans across asynchronous tasks or crosses microservice boundaries.

In typical usage, OTEL instrumentation libraries handle context propagation automatically. However, there are cases where you need to do it manually, especially in asynchronous workflows or custom instrumentation.

Propagation with Async Functions

When dealing with Python async/await code, you can manually pass context if an automated instrumentation doesn’t handle it, or if you have custom logic. The steps are:

  1. Extract the current context (e.g., from an incoming HTTP request).

  2. Create a new span as a child of that context.

  3. Pass or embed the context into the async function so it can be reused.

import asyncio
from opentelemetry import trace
from opentelemetry.context import attach, detach, get_current

tracer = trace.get_tracer(__name__)

async def async_func(ctx):
    token = attach(ctx)
    try:
        current_span = trace.get_current_span()
        current_span.set_attribute("input.value", User Input)
        await asyncio.sleep(1)  # Simulate async work
    finally:
        detach(token)

def sync_func():
    with tracer.start_as_current_span("sync_span") as span:
        # Capture the current context 
        context = get_current()
        # Run the async function, passing the context
        asyncio.run(async_func(context))

if __name__ == "__main__":
    sync_func()

Propagation Across Different Microservices

When making HTTP or gRPC calls to another microservice, we typically propagate the current tracing context through HTTP headers. If you’re using the built-in instrumentation (like opentelemetry-instrumentation-requests or opentelemetry-instrumentation-httpx), it’s handled automatically. For a custom approach, you do the following:

  1. Inject the current span context into HTTP headers before sending the request.

  2. On the receiving microservice, extract the context from the incoming headers.

Example: Service A sends a request to Service B.

Service A:

import requests
from opentelemetry import trace
from opentelemetry.context import Context
from opentelemetry.propagators.textmap import CarrierT, DefaultTextMapPropagator

tracer = trace.get_tracer(__name__)

def make_request_to_service_b():
    # Start a new span for this operation
    with tracer.start_as_current_span("llm_service_a") as span:
        # Prepare headers
        headers = {}
        DefaultTextMapPropagator().inject(carrier=headers)  # Inject the current context

        # Make the request with the injected headers
        response = requests.get("http://service-b:5000/endpoint", headers=headers)
        return response.text

Service B:

from flask import Flask, request
from opentelemetry import trace
from opentelemetry.propagators.textmap import DefaultTextMapPropagator

app = Flask(__name__)
tracer = trace.get_tracer(__name__)

@app.route("/endpoint")
def endpoint():
    # Extract the context from incoming request
    context = DefaultTextMapPropagator().extract(carrier=dict(request.headers))

    # Create a new span as child
    with tracer.start_as_current_span("service_b_processing", context=context) as span:
        span.add_event("Received request in service B")
        # ... do some processing ...
        return "Hello from Service B"

Propagation with Concurrent Threads

When you submit tasks to a ThreadPoolExecutor or any other concurrency mechanism, each task runs in a separate thread. If you rely on a tracer’s current context (which stores the active span or baggage), it won’t automatically follow your tasks to those worker threads. By manually capturing the context in the main thread and then attaching it in each worker thread, you preserve the association between the tasks and the original trace context.

Example

Below is a detailed, annotated example to show how you can:

  1. Capture the current context before submitting tasks to the executor.

  2. Attach that context within each worker thread (using attach).

  3. Run your task logic (e.g., processing questions).

  4. Detach the context when the task is complete (using detach).

import concurrent.futures
from opentelemetry import trace
from opentelemetry.context import attach, detach, get_current

tracer = trace.get_tracer(__name__)

def func1():
    """
    Some example work done in a thread.
    """
    current_span = trace.get_current_span()
    current_span.set_attribute("input.value", User Input)
    return "func1 result"

def func2():
    """
    Another example function that logs an event to the current span.
    """
    current_span = trace.get_current_span()
    current_span.set_attribute("input.value", User Input2)
    return "func2 result"
    
    
def wrapped_func(func: Callable):
    """
    Demonstrates how to capture the current context in the main thread,
    and attach/detach it within each worker thread. 
    Wraps the original function to attach/detach the current context
    so the worker thread has the correct span context.
    """
    # Capture the context from the current thread
    main_context = get_current()
    def wrapper():
        token = attach(main_context)  # Attach context to this thread
        try:
            return func()
        finally:
            detach(token)              # Detach after finishing
    return wrapper
# Create a list of functions to be executed in parallel
funcs = [func1, func2, func1, func2]

with concurrent.futures.ThreadPoolExecutor() as executor:
 # Map each function to its wrapped version
 results = list(executor.map(lambda f: wrapped_func(f)(), funcs))

return results

2. Creating Custom Decorators

Decorators are a convenient way to instrument functions and methods across your codebase without having to insert tracing calls repeatedly. A custom decorator can:

  • Start a new span before the function call.

  • Add attributes/events with function arguments (inputs).

  • Return the function’s result (outputs) and annotate or log it in the span.

  • End the span.

Example Decorator Implementation:

from opentelemetry import trace

def trace_function(span_kind=None, additional_attributes=None):
    tracer = trace.get_tracer(__name__)
    def wrapper(*args, **kwargs):
        with tracer.start_as_current_span(func.__name__) as span:
               if span_kind:
                   span.set_attribute("openinference.span.kind", span_kind)

               span.set_attribute("input.value", str(args))

               if additional_attributes:
                   for key, value in additional_attributes.items():
                       span.set_attribute(key, value)
              
               result = func(*args, **kwargs)
               span.set_attribute("output.value", str(result))
               return result
    return wrapper

# Example Implementation
@trace_function(span_kind="LLM", additional_attributes={"llm.model_name": "gpt-4o"})
def process_text(text):
    return text.upper()

3. Filtering Spans Based on Specific Attributes

In large-scale applications, you may not need to record every single span. Instead, you might want to selectively sample:

  • Spans of a particular service or component.

  • Spans that meet certain business criteria (e.g., user.id in a specific subset).

  • Only error or slow spans.

By creating a custom sampler, you can dynamically control which spans get recorded/exported based on their attributes or names. This approach helps control telemetry volume and cost, while ensuring you capture the traces most relevant for debugging or analysis.

Custom Sampling Basics

Sampler Interface

In OTEL Python, you create a custom sampler by subclassing the Sampler interface from opentelemetry.sdk.trace.sampling. You then implement:

  1. should_sample(...)

    • Decides whether the span is recorded (Sampled) or dropped (NotSampled).

    • You can look at the attributes, span name, span kind, parent context, etc.

Sampling Result

When implementing should_sample, you must return a SamplingResult, which indicates:

  • Sampling Decision: Decision.RECORD_AND_SAMPLE, Decision.RECORD_ONLY, or Decision.DROP.

  • Attributes: You can optionally modify or add attributes in the returned SamplingResult (e.g., a reason for sampling).

Example:

Let's create an example sampling mechanism where spans with a specific user ID is dropped.

from opentelemetry.context import Context
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from opentelemetry.sdk.trace.sampling import Sampler, SamplingResult, Decision
from opentelemetry import trace

class UserBasedSampler(Sampler):
    """
    A custom sampler that drops any span having a `user.id` attribute matching
    a specified user ID. Otherwise, spans are recorded and sampled.
    """
    def should_sample(
        self,
        parent_context: Context,
        trace_id: int,
        name: str,
        kind,
        attributes: dict,
        links
    ) -> SamplingResult:
        user_id = attributes.get("user.id")
        if user_id == USER_ID_TO_DROP:
            # If this user matches the one we want to drop, do not sample
            return SamplingResult(
                decision=Decision.DROP,
                attributes={"sampler.reason": f"Dropping span for user.id={user_id}"}
            )
        else:
            # Otherwise, record and sample normally
            return SamplingResult(
                decision=Decision.RECORD_AND_SAMPLE,
                attributes={}
            )

You then pass your custom sampler into your tracer provider.

tracer_provider = TracerProvider(sampler=UserBasedSampler())

4. Inheriting Context Attributes in Manual Spans

When using OpenTelemetry, context attributes are not automatically attached to spans created manually using start_as_current_span. This differs from auto-instrumentation (e.g., _llm_context), where attributes from the context are explicitly merged into the span during its creation.

To ensure context attributes are available in manual spans, you must explicitly retrieve and set them:

# Solution: Custom Span Creation Helper

def create_span_with_context(tracer, name, **kwargs):
    with tracer.start_as_current_span(name, **kwargs) as span:
        context_attributes = dict(get_attributes_from_context())
        span.set_attributes(context_attributes)
        return span

This pattern ensures that all relevant context (like session IDs, user info, etc.) is inherited by manually created spans, preserving traceability and observability.

# example usage
with using_session("my-session-id"):
    with create_span_with_context(tracer, "my-manual-span") as span:
        # ... your code here

Last updated 17 days ago

Was this helpful?

🔭