Search
⌃K
Links

Large Language Models (LLM)

How to log your model schema for LLM use cases

LLM Model Overview

LLM Use Cases
Model Type
Expected Fields
Summarization
ModelTypes.GENERATIVE_LLM
prompt_column_name , response_column_name
Question Answering
ModelTypes.GENERATIVE_LLM
prompt_column_name, response_column_name

Step-by-Step Guide To Upload LLMs

Logging LLM Code Example

Python Pandas
Python Single Record

Example Row

document
summary
reference_summary
document_vector
summary_vector
ScareBLEU
rogueL
prediction_id
prediction_ts
It was once the edgiest place in town, attracting the ...
The Variety Theater in Cleveland, once a ...
Cleveland's Variety Theater was a renowned rock ...
[-0.0051908623, -0.05508642, -0.28958365, -0.2...
[-0.18691051, -0.2092136, -0.16603243, -0.3258...
5.390328
0.216561
dd824bd3-2097...
2023-04-05 20:33:22.006650000
from arize.pandas.logger import Client, Schema
from arize.utils.types import ModelTypes, Environments, EmbeddingColumnNames
​
API_KEY = 'ARIZE_API_KEY'
SPACE_KEY = 'YOUR SPACE KEY'
arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)
​
# Declare embedding feature columns
prompt_columns=EmbeddingColumnNames(
vector_column_name="document_vector",
data_column_name="document"
),
response_columns=EmbeddingColumnNames(
vector_column_name="summary_vector",
data_column_name="summary"
)
​
# Defina the Schema, including embedding information
schema = Schema(
prediction_id_column_name="prediction_id",
timestamp_column_name="prediction_ts",
tag_column_names=["sacreBLEU", "rouge1", "rouge2", "rougeL", "rougeLsum"],
prompt_column_names=prompt_columns,
response_column_names=response_columns
)
​
# Log the dataframe with the schema mapping
response = arize_client.log(
model_id=demo-generative-ai-text-summarization,
model_version= "v1",
model_type=ModelTypes.GENERATIVE_LLM,
environment=Environments.PRODUCTION,
dataframe=test_dataframe,
schema=schema,
)

Generate Evaluation Metrics (Optional)

Use Arize to generate LLM evaluation metrics using the Python SDK. Supported metrics include BLEU, SacreBLEU, GoogleBLEU, ROUGE, and METEOR.
To get access to these metrics, install the extra dependencies via
pip install arize[LLM_Evaluation]
Learn how to generate LLM evaluation metrics here.

Generate Embeddings Vectors (Optional)

GENERATIVE_LLM models require prompt_column_names and response_column_names using the EmbeddingColumnNames class to send the raw text and the embedding vector associated with it.
If you don't have embedding vectors yet, import the EmbeddingGenerator to generate your embeddings. Learn more here. If you already have your own embeddings, you can skip this.
pip install arize[AutoEmbeddings]
​
# Generate embeddings vectors
from arize.pandas.embeddings import EmbeddingGenerator, UseCases
​
generator = EmbeddingGenerator.from_use_case(
use_case=UseCases.NLP.SUMMARIZATION,
model_name="distilbert-base-uncased",
tokenizer_max_length=512,
batch_size=100
)
df["document_vector"] = generator.generate_embeddings(text_col=df["document"])
df["summary_vector"] = generator.generate_embeddings(text_col=df["summary"])

Prompt & Response Embeddings

The above example demonstrates how to obtain Arize-generated embedding vectors. Use the EmbeddingColumnNames object to define the prompt_column_names and response_column_names required in your model schema.
  • The vector_column_name should be the name of the column where the embedding vectors are stored. The embedding vector is the dense vector representation of the unstructured input.
    ⚠
    Note: embedding features are not sparse vectors.
  • The data_column_name should be the name of the column where the raw text associated with the vector is stored. It is the field typically chosen for NLP use cases. The column can contain both strings (full sentences) or a list of strings (token arrays).
# Declare embedding feature columns
prompt_columns=EmbeddingColumnNames(
vector_column_name="document_vector",
data_column_name="document"
),
response_columns=EmbeddingColumnNames(
vector_column_name="summary_vector",
data_column_name="summary"
)
See here for more information on embeddings and options for generating them.

Ingesting Prompt Template (Optional)

Arize supports ingesting prompt versions and the overall prompt to be natively tracked in platform.
PromptTemplateColumnNames: This group together the prompt templates with their version.
  • template_column_name: This is the column that contains the prompt template in string format. The variables are represented by using the double key braces: {{variable_name}}.
    • For example, your prompt template might look something like:
    Given the context of '{{retrieval_text_0}} + {{retrieval_text_1}}', and based on the frequently asked questions from our users, answer the user query as follows: '{{user_query}}'. Follow the instructions here exactly: '{{instruction}}'.
  • template_version_column_name: This is the column that contains the version of the template used. This will allow you to filter by this field in the Arize platform.
# Declare prompt template columns
prompt_template_columns = PromptTemplateColumnNames(
template_column_name="prompt_template",
template_version_column_name="prompt_template_name"
)

Ingesting LLM Configuration Parameters (Optional)

Arize supports the original LLM configurations to be tracked + monitored as well as modified as a part of the prompt playground feature. Currently, Open AI models are supported, with more model support coming soon.
LLMConfigColumnNames: This field groups together the LLM used and the hyper parameters passed to it.
  • model_column_name: Name of the column containing the names of the LLMs used to produce responses to the prompts. Typical examples are gpt-3.5turbo or gpt-4.
  • params_column_name: Name of column containing the hyperparameters used to configure the LLM used. The contents of the column must be well formatted JSON string. For example: {'max_tokens': 500, 'presence_penalty': 0.66, 'temperature': 0.28}
# Declare LLM config columns
llm_config_columns = LLMConfigColumnNames(
model_column_name="llm_config_model_name",
params_column_name="llm_params",
)
from arize.api import Client
from arize.utils.types import ModelTypes, Environments, Embedding
​
API_KEY = 'ARIZE_API_KEY'
SPACE_KEY = 'YOUR SPACE KEY'
arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)
​
# example features
# features = {
'feature_name': feature_value,
...
}
​
# example tags
tags = {
'step': 0,
'task_type': 'Q&A',
'conversation_id': '0a9c',
'api_call_duration': 6.238615,
'response_len': 73,
'prompt_len': 88,
'prompt_template': "You are an agent engineered to provide concise and precise answers to any question."
}
​
prompt = Embedding(
vector= [0.10031, -0.06012, -0.00259, -0.08568, ...],
data="! Why does the UI have to be so clunky and unintuitive?! It's like every time I try to navigate through the options, I end up getting lost in a sea of confusing menus and poorly labeled buttons. And don't even get me started on the color scheme - who thought neon green on black was a good idea? At this point, I'm ready to throw my computer out the window and switch to a different product altogether. Come on, developers - make things simpler for us already!"
)
response = Embedding(
vector= [-0.11366, -0.20642, -0.03115, -0.18099,...],
data="I'm sorry to hear that you're experiencing difficulties with the UI. As an AI language model, I don't have the capability to change the UI, but I suggest looking into user guides, tutorials or support forums to help navigate through the menus more easily. Additionally, you could provide feedback to the developers on the UI, either through in-product feedback tools or their website. They may take into account your suggestions in future updates."
)
​
# Log data into the Arize platform
response = arize_client.log(
prediction_id='plED4eERDCasd9797ca34',
model_id='sample-model-1',
model_type=ModelTypes.GENERATIVE_LLM,
environment=Environments.PRODUCTION,
model_version='v1',
prediction_timestamp=1618590882,
# prediction_label will default to 1 if not sent
actual_label=1, # 1 represents a thumbs up, the user liked the response of the model
features=features,
tags=tags,
prompt=prompt,
response=response,
)

Prompt & Response Embeddings

Use the Embedding object to create the prompt(containing the document_vector) and the response(containing the summary_vector). Both prompt and response are required for GENERATIVE_LLM model type.
  • The embedding vector is the dense vector representation of the unstructured input.
    ⚠
    Note: embedding features are not sparse vectors.
  • The embedding data is the raw text data associated with the vector.
# Build prompt & response embeddings
prompt = Embedding(
vector= [0.10031, -0.06012, -0.00259, -0.08568, ...],
data="! Why does the UI have to be so clunky and unintuitive?! It's like every time I try to navigate through the options, I end up getting lost in a sea of confusing menus and poorly labeled buttons. And don't even get me started on the color scheme - who thought neon green on black was a good idea? At this point, I'm ready to throw my computer out the window and switch to a different product altogether. Come on, developers - make things simpler for us already!"
)
response = Embedding(
vector= [-0.11366, -0.20642, -0.03115, -0.18099,...],
data="I'm sorry to hear that you're experiencing difficulties with the UI. As an AI language model, I don't have the capability to change the UI, but I suggest looking into user guides, tutorials or support forums to help navigate through the menus more easily. Additionally, you could provide feedback to the developers on the UI, either through in-product feedback tools or their website. They may take into account your suggestions in future updates."
)
See here for more information on embeddings and options for generating them.