Embedding & Cluster Analyzer

What is an Embedding Projector ?

Embedding Projectors are a great tool in visualizing and interpreting embeddings. In order to do so, we have to apply an algorithm to reduce the dimensionality of the embeddings to 2D/3D. UMAP (Uniform Manifold Approximation and Projection) is a dimensionality reduction technique that belongs to the neighbor graph category. Arize uses it to create a lower dimensional representation (2D or 3D) of your dataset, represented by embedding vectors.

Learn more about UMAP in comparison to other neighbor graph algorithms here.

How to use UMAP?

1. Select your data and Generate UMAP

   • Select a point on the drift visualization at the top of the page and generate UMAP to visualize the point in time selected

   

2. Investigate your worst drifted clusters

   • Clusters are groups of related points in the point cloud. The closer points or clusters are to each other, the more similar they are. Clusters allow you to easily visualize data that differs from your baseline but also allows you to visualize the global and local structure of your data.

   

   You can select a specific cluster to further investigate and view the data associated with the cluster
3. Investigate the data points that belong in that cluster

   

   When you select a cluster, Arize surfaces the data associated (shown on the right) and you can further investigate any data point by clicking "View Details"
4. Colorize/Filter your data

   UMAP enables users to identify patterns or the structure in the data to explain where model can be improved by applying colorizations and filters

How to Colorize and Filter Points in UMAP

Users can colorize and filter the UMAP visualization:

• By Dataset: points will be colored based on if they belong to the baseline or primary dataset.

• By Prediction Label/Score: points will be colored according to the prediction label/score obtained from the model.

• By Actual Label/Score: points will be colored according to the actual label/score obtained from the model.

• By Correctness: points will be colored based on whether or not the prediction was correct (i.e., does the prediction label match the actual label).

• By Confusion Matrix: after selecting a positive class, points will be colored by their confusion matrix value (true positive, true negative, false positive, or false negative).

• By Tag: identify patterns or insights in slices of data by choosing to color by tag.

• By Feature: Identify patterns or insights in slices of data by choosing to color by feature.

How to Configure your UMAP Generation?

Users can configure their UMAP generation by these parameters:

• Dimensions: choose between a 2D or a 3D plot.

• nNeighbors: controls how UMAP balances local versus global structure in the data. More specifically, it controls the definition of the local region, i.e., how many neighbors UMAP will look at to define a local region. It balances the focus from global to local structure. The lower/higher the value of nNeighbors, the more focus we put on the local/global structure of the dataset. Allowed values range from 5 to 100. Learn more here.

• minDist: provides the minimum distance apart that points are allowed to be in the low dimensional representation. Allowed values range from 0.0 to 0.99. Learn more here.

• Sample size: the number of points in the UMAP plot per dataset. For example, if you select 500 there will be 1000 points total in the plot. Allowed values range from 300 to 2500.

What is Clustering?

Clustering is the process of grouping similar data points together. The goal of clustering is to find patterns and structure in a data set and to divide the data points into groups, or clusters, that share certain characteristics.

How does Arize cluster the UMAP points?

Our clustering algorithm is an unsupervised learning technique, which means that it works on unlabelled data and finds patterns on its own.

How can clustering help me?

Clusters help you find patterns and structure in your dataset. Users are able to troubleshoot performance degradation by examining the underlaying data in the form of clusters and use these insights to improve your models performance.

Examples:

• you might realize that your model is confusing two classes that are similar (i.e sandals and sneakers)

• you have a cluster with a drift score close to -1, meaning that model is seeing production data that is unlike the training

The drift score measures the reference data coverage present in a given cluster or point cloud. A score of -1 means that the cluster only contains primary, or production, data. A score of 1 means that the cluster only contains only baseline data. A score of 0 means the cluster is equally composed of baseline and primary data. The white and blue bars represent the count in each dataset in that cluster.

In this example, the cluster contains 16 points in production, and 55 points in the baseline dataset.

Cluster Metrics

After choosing your desired cluster metric (e.g. euclidean distance, accuracy, custom metric, etc.), Arize automatically surfaces the clusters you should focus on for model improvement / troubleshooting so you can quickly find the root cause.

Arize automatically surfaces clusters of bad responses to focus on improvement

You can select the metric you want to use, and how you want the clusters to be sorted.

Select how to sort the clusters by evaluation score and dataset

How do I use clusters to improve my model?

Download Clusters Locally

Once a cluster that is impacting model performance has been identified, users can download the data in the cluster for active learning. This data includes all the information needed for labeling workflows. These clusters are highly focused groups of datapoints, enabling labeling teams to be more precise in their efforts.

Download cluster of datapoints

Example CSV

Export Data into Notebook Environment

Arize enables teams to continue analysis of their production data in notebooks.

With a few lines of Python code, users can export their data into Phoenix or a Jupyter notebook for further analysis.

Export to Phoenix

Learn more here.