Python

While the spans created via Arize/Phoenix and OpenInference create a solid foundation for tracing your application, sometimes you need to create and customize your LLM spans.

Google Colaboratory

Arize and OpenInference use the OpenTelemetry Trace API to create spans. Because Arize supports OpenTelemetry, this means that you can perform manual instrumentation, no LLM framework required! This guide will help you understand how to create and customize spans using the OpenTelemetry Trace API.

Prerequisites

Before you start, ensure you have the following tools and packages installed:

• Python 3.6 or higher

• OpenTelemetry API and SDK1

• OpenInference Semantic Conventions

pip install opentelemetry-api
pip install opentelemetry-sdk

Let's next install the OpenInference Semantic Conventions package so that we can construct spans with LLM semantic conventions:

pip install openinference-semantic-conventions

For full documentation on the OpenInference semantic conventions, please consult the specification https://arize-ai.github.io/openinference/spec/semantic_conventions.html

Configuring a Tracer

Configuring an OTel tracer involves some boilerplate code that the instrumentors in phoenix.trace take care of for you. If you're manually instrumenting your application, you'll need to implement this boilerplate yourself:

import openai
import opentelemetry
import pandas as pd
from openai import OpenAI
from openinference.instrumentation.openai import OpenAIInstrumentor
from openinference.semconv.trace import OpenInferenceSpanKindValues, SpanAttributes
from opentelemetry import trace as trace_api
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

# Set the Space and API keys as headers for authentication
headers = f"space_key={ARIZE_SPACE_KEY},api_key={ARIZE_API_KEY}"
os.environ['OTEL_EXPORTER_OTLP_TRACES_HEADERS'] = headers

# Set resource attributes for the name and version for your application
trace_attributes = {
    "model_id": "your model name",  # This is how your model will show up in Arize
    "model_version": "v1",  # You can filter your spans by model version in Arize
}
endpoint = "https://otlp.arize.com/v1"

span_exporter = OTLPSpanExporter(endpoint=endpoint)
tracer_provider = trace_sdk.TracerProvider(
    resource=Resource(attributes=trace_attributes)
)

tracer_provider.add_span_processor(
    SimpleSpanProcessor(
        span_exporter=OTLPSpanExporterGrpc(endpoint=ARIZE_PROD_ENDPOINT)
    )
)
trace_api.set_tracer_provider(tracer_provider=tracer_provider)

tracer = trace_api.get_tracer(__name__)

# Because we are using Open AI, we will use this along with our custom instrumentation
OpenAIInstrumentor().instrument()

This snippet contains a few OTel concepts:

• A resource represents an origin (e.g., a particular service, or in this case, a project) from which your spans are emitted.

• Span processors filter, batch, and perform operations on your spans prior to export.

• Your tracer provides a handle for you to create spans and add attributes in your application code.

• The collector (e.g., Phoenix) receives the spans exported by your application.

Creating spans

To create a span, you'll typically want it to be started as the current span.

def do_work():
    with tracer.start_as_current_span("span-name") as span:
        # do some work that 'span' will track
        print("doing some work...")
        # When the 'with' block goes out of scope, 'span' is closed for you

You can also use start_span to create a span without making it the current span. This is usually done to track concurrent or asynchronous operations.

Creating nested spans

If you have a distinct sub-operation you'd like to track as a part of another one, you can create span to represent the relationship:

def do_work():
    with tracer.start_as_current_span("parent") as parent:
        # do some work that 'parent' tracks
        print("doing some work...")
        # Create a nested span to track nested work
        with tracer.start_as_current_span("child") as child:
            # do some work that 'child' tracks
            print("doing some nested work...")
            # the nested span is closed when it's out of scope

        # This span is also closed when it goes out of scope

When you view spans in a trace visualization tool, child will be tracked as a nested span under parent.

Creating spans with decorators

It's common to have a single span track the execution of an entire function. In that scenario, there is a decorator you can use to reduce code:

@tracer.start_as_current_span("do_work")
def do_work():
    print("doing some work...")

Use of the decorator is equivalent to creating the span inside do_work() and ending it when do_work() is finished.

To use the decorator, you must have a tracer instance in scope for your function declaration.

If you need to add attributes or events then it's less convenient to use a decorator.

Get the current span

Sometimes it's helpful to access whatever the current span is at a point in time so that you can enrich it with more information.

from opentelemetry import trace

current_span = trace.get_current_span()
# enrich 'current_span' with some information

Add attributes to a span

Attributes let you attach key/value pairs to a spans so it carries more information about the current operation that it's tracking.

from opentelemetry import trace

current_span = trace.get_current_span()

current_span.set_attribute("operation.value", 1)
current_span.set_attribute("operation.name", "Saying hello!")
current_span.set_attribute("operation.other-stuff", [1, 2, 3])

Notice above that the attributes have a specific prefix operation. When adding custom attributes, it's best practice to vendor your attributes (e.x. mycompany.) so that your attributes do not clash with semantic conventions.

Add Semantic Attributes

Semantic attributes are pre-defined attributes that are well-known naming conventions for common kinds of data. Using semantic attributes lets you normalize this kind of information across your systems. In the case of Phoenix, the OpenInference Semantic Conventions package provides a set of well-known attributes that are used to represent LLM application specific semantic conventions.

To use OpenInference Semantic Attributes in Python, ensure you have the semantic conventions package:

pip install openinference-semantic-conventions

Setting attributes is crucial for understanding the flow of data and messages through your LLM application, which facilitates easier debugging and analysis. By setting attributes such as OUTPUT_VALUE and OUTPUT_MESSAGES, you can capture essential output details and interaction messages within the context of a span. This allows you to record the response and categorize and store messages exchanged by components in a structured format:

from openinference.semconv.trace import SpanAttributes

span.set_attribute(SpanAttributes.OUTPUT_VALUE, response)

# This shows up under `output_messages` tab on the span page within Arize
span.set_attribute(
    f"{SpanAttributes.LLM_OUTPUT_MESSAGES}.0.{MessageAttributes.MESSAGE_ROLE}",
    "assistant",
)
span.set_attribute(
    f"{SpanAttributes.LLM_OUTPUT_MESSAGES}.0.{MessageAttributes.MESSAGE_CONTENT}",
    response,
)

For more detailed information on semantic conventions, please refer to our Semantic Conventions section.

Add Context Attributes

Context attributes refer to span attributes that can be read from the OTEL Context. Learn more here. Our OpenInference core instrumentation package offers a convenient function, get_attributes_from_context, to read these context attributes from the OTEL Context.

Install the package via pip install openinference-instrumentation

In the following example, we assume the following are set in the OTEL context:

tags = ["tag_1", "tag_2"]
metadata = {
    "key-1": 1,
    "key-2": "2",
}
prompt_template = "Please describe the weather forecast for {city} on {date}"
prompt_template_variables = {"city": "Johannesburg", "date":"July 11"}
prompt_template_version = "v1.0"

See Set Context Attributes to learn how to set attributes in the OTEL context. We then use get_attributes_from_context to extract them from the OTEL context. You can use it in your manual instrumentation to attach these attributes to your spans.

from openinference.instrumentation import get_attributes_from_context

span.set_attributes(dict(get_attributes_from_context()))
# The span will then have the following attributes attached:
# {
#    'session.id': 'my-session-id',
#    'user.id': 'my-user-id',
#    'metadata': '{"key-1": 1, "key-2": "2"}',
#    'tag.tags': ['tag_1', 'tag_2'],
#    'llm.prompt_template.template': 'Please describe the weather forecast for {city} on {date}',
#    'llm.prompt_template.version': 'v1.0',
#    'llm.prompt_template.variables': '{"city": "Johannesburg", "date": "July 11"}'
# }

Adding events

Events are human-readable messages that represent "something happening" at a particular moment during the lifetime of a span. You can think of it as a primitive log.

from opentelemetry import trace

current_span = trace.get_current_span()

current_span.add_event("Gonna try it!")

# Do the thing

current_span.add_event("Did it!")

Set span status

The span status allows you to signal the success or failure of the code executed within the span.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

current_span = trace.get_current_span()

try:
    # something that might fail
except:
    current_span.set_status(Status(StatusCode.ERROR))

Record exceptions in spans

It can be a good idea to record exceptions when they happen. It’s recommended to do this in conjunction with setting span status.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

current_span = trace.get_current_span()

try:
    # something that might fail

# Consider catching a more specific exception in your code
except Exception as ex:
    current_span.set_status(Status(StatusCode.ERROR))
    current_span.record_exception(ex)