LLMRunMetadata

Ingest metadata about your LLM inferences

View source on Github

Arize class to map up to 4 columns: total_token_count_column_name , prompt_token_count_column_name, response_token_count_column_name, andresponse_latency_ms_column_name

class LLMRunMetadata:
    total_token_count: Optional[int] = None
    prompt_token_count: Optional[int] = None
    response_token_count: Optional[int] = None
    response_latency_ms: Optional[Union[int,float]] = None

Parameters	Data Type	Description
total_token_count	int	The total number of tokens used in the inference, both in the prompt sent to the LLM and in its response
promt_token_count	int	The number of tokens used in the prompt sent to the LLM
response_token_count	int	The number of tokens used in the response returned by the LLM
response_latency_ms	int or float	The latency (in ms) experienced during the LLM run

Code Example

from arize.utils.types import LLMRunMetadata

# Declare LLM run metadata
llm_run_metadata = LLMRunMetadata(
    total_token_count = 4325,
    prompt_token_count = 2325,
    response_token_count = 2000,
    response_latency_ms = 20000,
)