Image Classification

How to log your model schema for image classification models

Image Classification Model Overview

Image classification models take an image as input and return a predicted label for the image.

*all classification variant specifications apply to the Image Classification model type, with the addition of embeddings

Performance Metrics

Accuracy, Recall, Precision, FPR, FNR, F1, Sensitivity, Specificity

Code Example

The EmbeddingColumnNames class constructs your embedding objects. You can log them into the platform using a dictionary that maps the embedding feature names to the embedding objects. See our API reference for more details.

Navigate here for step-by-step instructions to view private AWS S3 image links.

Python Pandas

Example Row

image_vector	image_link	prediction_label	actual_label	prediction_score	actual_score	Timestamp
[1.0, 2, 3]	"https://link-to-my-image.png"	car	bus	0.3	1	1618590882

from arize.pandas.logger import Client, Schema
from arize.utils.types import ModelTypes, Environments, EmbeddingColumnNames

API_KEY = 'ARIZE_API_KEY'
SPACE_KEY = 'YOUR SPACE KEY'
arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)


# Declare which columns are the feature columns
feature_column_names=[
    "MERCHANT_TYPE", 
    "ENTRY_MODE", 
    "STATE", 
    "MEAN_AMOUNT", 
    "STD_AMOUNT", 
    "TX_AMOUNT",
]

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

# Declare embedding feature columns
embedding_feature_column_names = {
    # Dictionary keys will be the name of the embedding feature in the app
    "embedding_display_name": EmbeddingColumnNames(
        vector_column_name="image_vector",  # column name of the vectors, required
        link_to_data_column_name="image_link", # column name of the link to the images, optional
    )
}

# Defina the Schema, including embedding information
schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    prediction_label_column_name="PREDICTION",
    prediction_score_column_name="PREDICTION_SCORE",
    actual_label_column_name="ACTUAL",
    actual_score_column_name="ACTUAL_SCORE",
    feature_column_names=feature_column_names,
    embedding_feature_column_names=embedding_feature_column_names,
    tag_column_names=tag_columns,
)

# Log the dataframe with the schema mapping 
response = arize_client.log(
    model_id="sample-model-1",
    model_version= "v1",
    model_type=ModelTypes.SCORE_CATEGORICAL,
    environment=Environments.PRODUCTION,
    dataframe=test_dataframe,
    schema=schema,
)

Google Colaboratory

Image Classification Embedding Features

Arize supports logging the embedding features associated with the image the model is acting on and the image itself using the EmbeddingColumnNames object.

• The vector_column_name should be the name of the column where the embedding vectors are stored. The embedding vector is the dense vector representation of the unstructured input. ⚠️ Note: embedding features are not sparse vectors.

• The link_to_data_column_name should be the name of the column where the URL links to the source images, that your model classifies, are stored.

{ 
    "embedding_display_name": EmbeddingColumnNames(
        vector_column_name="image_vector", 
        link_to_data_column_name="image_link" 
    ) 
}

See here for more information on embeddings and options for generating them.

Python Single Record

from arize.api import Client
from arize.utils.types import ModelTypes, Environments, Embedding

API_KEY = 'ARIZE_API_KEY'
SPACE_KEY = 'YOUR SPACE KEY'
arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)

# Example features; features & tags can be optionally defined with typing
features = {
    'state': 'ca',
    'city': 'berkeley',
    'merchant_name': 'Peets Coffee',
    'pos_approved': TypedValue(value=False, type=ArizeTypes.INT),
    'item_count': 10,
    'merchant_type': 'coffee shop',
    'charge_amount': TypedValue(value=20.11, type=ArizeTypes.FLOAT),
}
    
# Example embedding features
embedding_features = {
    "image_embedding": Embedding(
        vector=np.array([1.0, 2, 3]),
        link_to_data="https://link-to-my-image.png",
    ),
}

# Log data into the Arize platform
response = arize.log(
    model_id='sample-model-1', 
    model_version='v1", 
    model_type=ModelTypes.SCORE_CATEGORICAL, 
    environment=Environments.PRODUCTION,
    features=features
    prediction_label="not fraud",
    prediction_score = 0.3
    actual_label="fraud",
    actual_score = 1
    features=features,
    embedding_features=embedding_features 
)

CV Embedding Features

Arize supports logging the embedding features associated with the image the model is acting on and the image itself using the Embedding object.

• The embedding vector is the dense vector representation of the unstructured input. ⚠️ Note: embedding features are not sparse vectors.

• The embedding link_to_data is used to pass URL links to the source image your model is classifying.

{ 
    "embedding_display_name": Embedding(
            vector=np.array([1.0, 2, 3]),
            link_to_data="https://link-to-my-image.png",
    ) 
}

See here for more information on embeddings and options for generating them.

UI Import JSON Input

When configuring an embedding in the UI using File Import

"embedding_features": [{
   "my_feature": // #required, my_feature is the name of the feature
        {
           vector: "vector_col", // #required, vector_col is the column name of the vector
           raw_data: "raw_data_col", // #optional
           link_to_data: "link_to_data_col" // #optional
        }
}]

Example file schema with embedding features

{
  "prediction_id": "prediction_id",
  "timestamp": "timestamp",
  "tags": "tag/",
  "prediction_score": "prediction_score",
  "prediction_label": "prediction_label",
  "actual_label": "actual_label",
  "actual_score": "actual_score",
  "shap_values": "shap/",
  "version": "version", // lookup the column "version" in the file
  "batch_id": "batch_id",
  "exclude": [
    "<column1 name>",
    "<column2 name>"
  ],
  "embedding_features": [
    {
      "embedding_1": {
        "vector": "vector_column_1"
        "raw_data": "raw_data_column_1",
        "link_to_data": "link_to_data_column"
      }
    }
  ]
}

Input for API

When configuring an embedding in the UI using the API

"embeddingFeatures": [{
   "featureName": "my_feature",
   "vectorCol": "vector_col",
   "rawDataCol": "raw_data_col",
   "linkToDataCol": "link_to_data_col"
}]

Example file schema with embedding features

prediction_id: prediction_id
timestamp: timestamp
features: feature/
tags: tag/
prediction_score: prediction_score
prediction_label: prediction_label
actual_label: actual_label
actual_score: actual_score
shap_values: shap/
version: version // lookup the column "version" in the file
batch_id: batch_id
exclude: // leave empty to omit column exclusions
embedding_features: // leave empty to omit embeddings