Tracing

Anatomy of a trace

A trace represents a single request, which is made up of multiple spans.

A trace records the paths taken by requests as they propagate through multiple steps. Traces make it easy to debug issues and understand the flow of your application.

A simplified view of a trace containing spans

Spans

A span represents a unit of work, with an input, output, start, and end time. It tracks specific operations such as a retriever, tool call, or LLM step.

Here is an example of an LLM span in JSON form below, with attributes such as the llm input messages and output value.

{
    "name": "llm",
    "context": {
        "trace_id": "ed7b336d-e71a-46f0-a334-5f2e87cb6cfc",
        "span_id": "ad67332a-38bd-428e-9f62-538ba2fa90d4"
    },
    "span_kind": "LLM",
    "parent_id": "f89ebb7c-10f6-4bf8-8a74-57324d2556ef",
    "start_time": "2023-09-07T12:54:47.597121-06:00",
    "end_time": "2023-09-07T12:54:49.321811-06:00",
    "status_code": "OK",
    "status_message": "",
    "attributes": {
        "llm.input_messages": [
            {
                "message.role": "system",
                "message.content": "You are an expert Q&A system that is trusted around the world.\nAlways answer the query using the provided context information, and not prior knowledge.\nSome rules to follow:\n1. Never directly reference the given context in your answer.\n2. Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines."
            },
            {
                "message.role": "user",
                "message.content": "Hello?"
            }
        ],
        "output.value": "assistant: Yes I am here",
        "output.mime_type": "text/plain"
    },
    "events": [],
}

Span Kinds

Every span has a type, which signifies the

Span Kind	Description
LLM	Call to an LLM for a completion or chat
Chain	The starting point and link between application steps
Tool	API or function invoked on behalf of an LLM
Agent	Root span which contains a set of LLM and tool invocations
Embedding	Encoding of unstructured data
Retriever	Data retrieval query for context from a datastore
Reranker	Relevance based re-ordering of documents

Span Attributes

Attributes are key-value pairs that contain metadata that you can use to annotate a span to carry information about the operation it is tracking.

For example, if a span invokes an LLM, you can capture the model name, the invocation parameters, the token count, and so on.

Attributes have the following rules:

• Keys must be non-null string values

• Values must be a non-null string, boolean, floating point value, integer, or an array of these values Additionally, there are Semantic Attributes, which are known naming conventions for metadata that is typically present in common operations. It's helpful to use semantic attribute naming wherever possible so that common kinds of metadata are standardized across systems. See semantic conventions for more information.

📚 Learn more about LLM Tracing.

How does tracing work?

There are five key components within tracing - Instrumentation, Span Processor, Exporter, Protocol, and Collector. We've built all of these within our tracing tools to make it easy to troubleshoot your application and surface issues.

Instrumentation

In order for an application to emit traces for analysis, the application must be instrumented. Your application can be manually or automatically instrumented.

Our auto-instrumentation plugins will collect spans for you with just a few lines of code. All of these instrumentors are managed via a single repository called OpenInference.

Instrumentation works by automatically detecting specific function calls and logging their attributes to span properties, such as input variables, outputs, latency, and more. Each auto-instrumentor is built specifically to map to the provider's SDK, such as OpenAI, Bedrock, Llamaindex, and Langchain.

SpanProcessor

Span processors dictate how the spans should be processed during the execution of a request, and whether it should be blocking or not. There are 2 main span processors:

• Batch Processor - Used for product deployments, this is non-blocking for your application. We default to batch when we register your instrumentation to reduce latency.

• SimpleSpanProcessor - This is a synchronous logging instrumentation where the log call is blocking, delays in collector will back up into your application. Used for debugging setups.

Read more about how this is setup on our arize-otel page.

Exporter

An exporter takes the spans created via instrumentation and exports them to a collector. In simple terms, it sends the data to an Arize URL with a specified format. The exporter used for Arize is the GRPC exporter.

OpenTelemetry Protocol

OpenTelemetetry Protocol (or OTLP for short) is the means by which traces arrive from your application to the Arize collector. Arize currently supports OTLP over GRPC.

Collector

The Arize server is a collector and a UI that helps you troubleshoot your application in real time. Arize receives your logs and visualizes them on our dashboard. We use many technologies under the hood to ensure we can support millions of traces and return you aggregations and insights on your data.

We receive spans at https://otlp.arize.com/v1 in the US and https://otlp.eu-west-1a.arize.com/v1 in Europe.