Concepts: Inferences
This section introduces inferences and schemas, the starting concepts needed to use Phoenix with inferences.
For comprehensive descriptions of
phoenix.Inferences
andphoenix.Schema
, see the API reference.For tips on creating your own Phoenix inferences and schemas, see the how-to guide.
Inferences
Phoenix inferences are an instance of phoenix.Inferences
that contains three pieces of information:
The data itself (a pandas dataframe)
A name that appears in the UI
For example, if you have a dataframe prod_df
that is described by a schema prod_schema
, you can define inferences prod_ds
with
If you launch Phoenix with these inferences, you will see inferences named "production" in the UI.
How many inferences do I need?
You can launch Phoenix with zero, one, or two sets of inferences.
With no inferences, Phoenix runs in the background and collects trace data emitted by your instrumented LLM application. With a single inference set, Phoenix provides insights into model performance and data quality. With two inference sets, Phoenix compares your inferences and gives insights into drift in addition to model performance and data quality, or helps you debug your retrieval-augmented generation applications.
Use Zero Inference sets When:
You want to run Phoenix in the background to collect trace data from your instrumented LLM application.
Use a Single Inference set When:
You have only a single cohort of data, e.g., only training data.
You care about model performance and data quality, but not drift.
Use Two Inference sets When:
You want to compare cohorts of data, e.g., training vs. production.
You care about drift in addition to model performance and data quality.
Which inference set is which?
Your reference inferences provides a baseline against which to compare your primary inferences.
To compare two inference sets with Phoenix, you must select one inference set as primary and one to serve as a reference. As the name suggests, your primary inference set contains the data you care about most, perhaps because your model's performance on this data directly affects your customers or users. Your reference inferences, in contrast, is usually of secondary importance and serves as a baseline against which to compare your primary inferences.
Very often, your primary inferences will contain production data and your reference inferences will contain training data. However, that's not always the case; you can imagine a scenario where you want to check your test set for drift relative to your training data, or use your test set as a baseline against which to compare your production data. When choosing primary and reference inference sets, it matters less where your data comes from than how important the data is and what role the data serves relative to your other data.
Corpus Inference set (Information Retrieval)
The only difference for the corpus inferences is that it needs a separate schema because it have a different set of columns compared to the model data. See the schema section for more details.
Schemas
A Phoenix schema is an instance of phoenix.Schema
that maps the columns of your dataframe to fields that Phoenix expects and understands. Use your schema to tell Phoenix what the data in your dataframe means.
For example, if you have a dataframe containing Fisher's Iris data that looks like this:
7.7
3.0
6.1
2.3
virginica
versicolor
5.4
3.9
1.7
0.4
setosa
setosa
6.3
3.3
4.7
1.6
versicolor
versicolor
6.2
3.4
5.4
2.3
virginica
setosa
5.8
2.7
5.1
1.9
virginica
virginica
your schema might look like this:
How many schemas do I need?
Usually one, sometimes two.
Each inference set needs a schema. If your primary and reference inferences have the same format, then you only need one schema. For example, if you have dataframes train_df
and prod_df
that share an identical format described by a schema named schema
, then you can define inference sets train_ds
and prod_ds
with
Sometimes, you'll encounter scenarios where the formats of your primary and reference inference sets differ. For example, you'll need two schemas if:
Your production data has timestamps indicating the time at which an inference was made, but your training data does not.
Your training data has ground truth (what we call actuals in Phoenix nomenclature), but your production data does not.
A new version of your model has a differing set of features from a previous version.
In cases like these, you'll need to define two schemas, one for each inference set. For example, if you have dataframes train_df
and prod_df
that are described by schemas train_schema
and prod_schema
, respectively, then you can define inference sets train_ds
and prod_ds
with
Schema for Corpus Inferences (Information Retrieval)
A corpus inference set, containing documents for information retrieval, typically has a different set of columns than those found in the model data from either production or training, and requires a separate schema. Below is an example schema for a corpus inference set with three columns: the id
, text
, and embedding
for each document in the corpus.
Last updated
Was this helpful?