Capture Feedback
feedback and annotations are available for arize-phoenix>=4.20.0 and are in beta
When building LLM applications, it is important to collect feedback to understand how your app is performing in production. The ability to observe user feedback along with traces can be very powerful as it allows you to drill down into the most interesting examples. Once you have identified these example, you can share them for further review, automatic evaluation, or fine-tuning.
Phoenix lets you attach user feedback to spans and traces in the form of annotations. It's helpful to expose a simple mechanism (such as 👍👎) to collect user feedback in your app. You can then use the Phoenix API to attach feedback to a span.
Phoenix expects feedback to be in the form of an annotation. Annotations consist of these fields:
Note that you can provide a label, a score, or both. With Phoenix an annotation has a name (like correctness), is associated with an annotator (either an LLM or a HUMAN) and can be attached to the spans you have logged to Phoenix.
Send Annotations to Phoenix
Once you construct the annotation, you can send this to Phoenix via it's REST API. You can POST an annotation from your application to /v1/span_annotations
like so:
Retrieve the current span_id
If you'd like to collect feedback on currently instrumented code, you can get the current span using the opentelemetry
SDK.
You can use the span_id to send an annotation associated with that span.
Annotate Traces in the UI
Phoenix also allows you to manually annotate traces with feedback within the application. This can be useful for adding context to a trace, such as a user's comment or a note about a specific issue. You can annotate a span directly from the span details view.
Last updated