feedback and annotations are available for arize-phoenix>=4.20.0
When building LLM applications, it is important to collect feedback to understand how your app is performing in production. The ability to observe user feedback along with traces can be very powerful as it allows you to drill down into the most interesting examples. Once you have identified these example, you can share them for further review, automatic evaluation, or fine-tuning.
Phoenix lets you attach user feedback to spans and traces in the form of annotations. It's helpful to expose a simple mechanism (such as 👍👎) to collect user feedback in your app. You can then use the Phoenix API to attach feedback to a span.
Phoenix expects feedback to be in the form of an annotation. Annotations consist of these fields:
{
"span_id": "67f6740bbe1ddc3f", // the id of the span to annotate
"name": "correctness", // the name of your annotator
"annotator_kind": "HUMAN", // HUMAN or LLM
"result": {
"label": "correct", // A human-readable category for the feedback
"score": 1, // a numeric score, can be 0 or 1, or a range like 0 to 100
"explanation": "The response answered the question I asked"
}
}
Note that you can provide a label, a score, or both. With Phoenix an annotation has a name (like correctness), is associated with an annotator (either an LLM or a HUMAN) and can be attached to the spans you have logged to Phoenix.
Send Annotations to Phoenix
Once you construct the annotation, you can send this to Phoenix via it's REST API. You can POST an annotation from your application to /v1/span_annotations like so:
If you're self-hosting Phoenix, be sure to change the endpoint in the code below to <your phoenix endpoint>/v1/span_annotations?sync=false
Retrieve the current span_id
If you'd like to collect feedback on currently instrumented code, you can get the current span using the opentelemetry SDK.
For LangChain, import get_current_span from our instrumentation library instead.
from opentelemetry.trace import format_span_id
from openinference.instrumentation.langchain import get_current_span
span = get_current_span()
if span is not None:
span_id = format_span_id(span.get_span_context().span_id)
You can use the span_id to send an annotation associated with that span.
Phoenix also allows you to manually annotate traces with feedback within the application. This can be useful for adding context to a trace, such as a user's comment or a note about a specific issue. You can annotate a span directly from the span details view.
