Large Language Models (LLM)
How to log your model schema for LLM use cases
LLM Use Cases | Model Type | Expected Fields |
|---|---|---|
Summarization | ModelTypes.GENERATIVE_LLM | prompt_column_name , response_column_name |
Question Answering | ModelTypes.GENERATIVE_LLM | prompt_column_name, response_column_name |
Python Pandas
Python Single Record
document | summary | reference_summary | document_vector | summary_vector | ScareBLEU | rogueL | prediction_id | prediction_ts |
|---|---|---|---|---|---|---|---|---|
It was once the edgiest place in town, attracting the ... | The Variety Theater in Cleveland, once a ... | Cleveland's Variety Theater was a renowned rock ... | [-0.0051908623, -0.05508642, -0.28958365, -0.2... | [-0.18691051, -0.2092136, -0.16603243, -0.3258... | 5.390328 | 0.216561 | dd824bd3-2097... | 2023-04-05 20:33:22.006650000 |
from arize.pandas.logger import Client, Schema
from arize.utils.types import ModelTypes, Environments, EmbeddingColumnNames
API_KEY = 'ARIZE_API_KEY'
SPACE_KEY = 'YOUR SPACE KEY'
arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)
# Declare embedding feature columns
prompt_columns=EmbeddingColumnNames(
vector_column_name="document_vector",
data_column_name="document"
),
response_columns=EmbeddingColumnNames(
vector_column_name="summary_vector",
data_column_name="summary"
)
# Defina the Schema, including embedding information
schema = Schema(
prediction_id_column_name="prediction_id",
timestamp_column_name="prediction_ts",
tag_column_names=["sacreBLEU", "rouge1", "rouge2", "rougeL", "rougeLsum"],
prompt_column_names=prompt_columns,
response_column_names=response_columns
)
# Log the dataframe with the schema mapping
response = arize_client.log(
model_id=demo-generative-ai-text-summarization,
model_version= "v1",
model_type=ModelTypes.GENERATIVE_LLM,
environment=Environments.PRODUCTION,
dataframe=test_dataframe,
schema=schema,
)
Use Arize to generate LLM evaluation metrics using the Python SDK. Supported metrics include BLEU, SacreBLEU, GoogleBLEU, ROUGE, and METEOR.
To get access to these metrics, install the extra dependencies via
pip install arize[LLM_Evaluation]
GENERATIVE_LLM models require prompt_column_names and response_column_names using the EmbeddingColumnNames class to send the raw text and the embedding vector associated with it. If you don't have embedding vectors yet, import the EmbeddingGenerator to generate your embeddings. Learn more here. If you already have your own embeddings, you can skip this.
pip install arize[AutoEmbeddings]
# Generate embeddings vectors
from arize.pandas.embeddings import EmbeddingGenerator, UseCases
generator = EmbeddingGenerator.from_use_case(
use_case=UseCases.NLP.SUMMARIZATION,
model_name="distilbert-base-uncased",
tokenizer_max_length=512,
batch_size=100
)
df["document_vector"] = generator.generate_embeddings(text_col=df["document"])
df["summary_vector"] = generator.generate_embeddings(text_col=df["summary"])
The above example demonstrates how to obtain Arize-generated embedding vectors. Use the
EmbeddingColumnNames object to define the prompt_column_names and response_column_names required in your model schema. - The
vector_column_nameshould be the name of the column where the embedding vectors are stored. The embedding vector is the dense vector representation of the unstructured input.Note: embedding features are not sparse vectors.⚠ - The
data_column_nameshould be the name of the column where the raw text associated with the vector is stored. It is the field typically chosen for NLP use cases. The column can contain both strings (full sentences) or a list of strings (token arrays).
# Declare embedding feature columns
prompt_columns=EmbeddingColumnNames(
vector_column_name="document_vector",
data_column_name="document"
),
response_columns=EmbeddingColumnNames(
vector_column_name="summary_vector",
data_column_name="summary"
)
Arize supports ingesting prompt versions and the overall prompt to be natively tracked in platform.
PromptTemplateColumnNames: This group together the prompt templates with their version.template_column_name: This is the column that contains the prompt template in string format. The variables are represented by using the double key braces:{{variable_name}}.- For example, your prompt template might look something like:
Given the context of '{{retrieval_text_0}} + {{retrieval_text_1}}', and based on the frequently asked questions from our users, answer the user query as follows: '{{user_query}}'. Follow the instructions here exactly: '{{instruction}}'.template_version_column_name: This is the column that contains the version of the template used. This will allow you to filter by this field in the Arize platform.
# Declare prompt template columns
prompt_template_columns = PromptTemplateColumnNames(
template_column_name="prompt_template",
template_version_column_name="prompt_template_name"
)
Arize supports the original LLM configurations to be tracked + monitored as well as modified as a part of the prompt playground feature. Currently, Open AI models are supported, with more model support coming soon.
LLMConfigColumnNames: This field groups together the LLM used and the hyper parameters passed to it.model_column_name: Name of the column containing the names of the LLMs used to produce responses to the prompts. Typical examples aregpt-3.5turboorgpt-4.params_column_name: Name of column containing the hyperparameters used to configure the LLM used. The contents of the column must be well formatted JSON string. For example:{'max_tokens': 500, 'presence_penalty': 0.66, 'temperature': 0.28}
# Declare LLM config columns
llm_config_columns = LLMConfigColumnNames(
model_column_name="llm_config_model_name",
params_column_name="llm_params",
)
from arize.api import Client
from arize.utils.types import ModelTypes, Environments, Embedding
API_KEY = 'ARIZE_API_KEY'
SPACE_KEY = 'YOUR SPACE KEY'
arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)
# example features
# features = {
'feature_name': feature_value,
...
}
# example tags
tags = {
'step': 0,
'task_type': 'Q&A',
'conversation_id': '0a9c',
'api_call_duration': 6.238615,
'response_len': 73,
'prompt_len': 88,
'prompt_template': "You are an agent engineered to provide concise and precise answers to any question."
}
prompt = Embedding(
vector= [0.10031, -0.06012, -0.00259, -0.08568, ...],
data="! Why does the UI have to be so clunky and unintuitive?! It's like every time I try to navigate through the options, I end up getting lost in a sea of confusing menus and poorly labeled buttons. And don't even get me started on the color scheme - who thought neon green on black was a good idea? At this point, I'm ready to throw my computer out the window and switch to a different product altogether. Come on, developers - make things simpler for us already!"
)
response = Embedding(
vector= [-0.11366, -0.20642, -0.03115, -0.18099,...],
data="I'm sorry to hear that you're experiencing difficulties with the UI. As an AI language model, I don't have the capability to change the UI, but I suggest looking into user guides, tutorials or support forums to help navigate through the menus more easily. Additionally, you could provide feedback to the developers on the UI, either through in-product feedback tools or their website. They may take into account your suggestions in future updates."
)
# Log data into the Arize platform
response = arize_client.log(
prediction_id='plED4eERDCasd9797ca34',
model_id='sample-model-1',
model_type=ModelTypes.GENERATIVE_LLM,
environment=Environments.PRODUCTION,
model_version='v1',
prediction_timestamp=1618590882,
# prediction_label will default to 1 if not sent
actual_label=1, # 1 represents a thumbs up, the user liked the response of the model
features=features,
tags=tags,
prompt=prompt,
response=response,
)
Use the
Embedding object to create the prompt(containing the document_vector) and the response(containing the summary_vector). Both prompt and response are required for GENERATIVE_LLM model type.- The embedding
vectoris the dense vector representation of the unstructured input.Note: embedding features are not sparse vectors.⚠ - The embedding
datais the raw text data associated with the vector.
# Build prompt & response embeddings
prompt = Embedding(
vector= [0.10031, -0.06012, -0.00259, -0.08568, ...],
data="! Why does the UI have to be so clunky and unintuitive?! It's like every time I try to navigate through the options, I end up getting lost in a sea of confusing menus and poorly labeled buttons. And don't even get me started on the color scheme - who thought neon green on black was a good idea? At this point, I'm ready to throw my computer out the window and switch to a different product altogether. Come on, developers - make things simpler for us already!"
)
response = Embedding(
vector= [-0.11366, -0.20642, -0.03115, -0.18099,...],
data="I'm sorry to hear that you're experiencing difficulties with the UI. As an AI language model, I don't have the capability to change the UI, but I suggest looking into user guides, tutorials or support forums to help navigate through the menus more easily. Additionally, you could provide feedback to the developers on the UI, either through in-product feedback tools or their website. They may take into account your suggestions in future updates."
)
Last modified 13d ago