Object Detection

How to declare your model schema and log data for object detection models

Object detection model type support is in early release. Reach out to [e-mail] for access.

Object Detection Model Overview

Object detection models identify and locate objects within images or videos by assigning them specific bounding boxes.

Applicable Metrics: Accuracy, Euclidian Distance (embeddings)

Click here for all valid model types and metric combinations.

Object Detection Code Example

Python Batch

Example Row

image_vector	image_link	prediction_bboxes	actual_bboxes	prediction_categories	actual_categories	actual_super_categories	prediction_scores	Timestamp
[0.24713118374347687, 0.7061651349067688, 1.12...	"https://link-to-my-image.png"	[[50.43, 109.49, 538.21...	[[55.39, 107.72, 539.25, 362.9], [554.41, 194....	[bus]	[bus, person, person]	[vehicle, person, person]	[0.9997552]	1618590882

embedding_feature_column_names={
    "image_embedding": EmbeddingColumnNames(
        vector_column_name="image_vector",
        link_to_data_column_name="url"
    )
}
object_detection_prediction_column_names=ObjectDetectionColumnNames(
    bounding_boxes_coordinates_column_name="prediction_bboxes",
    categories_column_name="prediction_categories",
    scores_column_name="prediction_scores"
)
object_detection_actual_column_names=ObjectDetectionColumnNames(
    bounding_boxes_coordinates_column_name="actual_bboxes",
    categories_column_name="actual_categories",
)

# feature & tag columns can be optionally defined with typing:
tag_columns = TypedColumns(
    inferred=["name"],
    to_int=["zip_code", "age"]
)

# Defina the Schema, including embedding information
schema = Schema(
    prediction_id_column_name="prediction_id",
    timestamp_column_name="prediction_ts",
    tag_column_names=tag_columns,
    embedding_feature_column_names=embedding_feature_column_names,
    object_detection_prediction_column_names=object_detection_prediction_column_names,
    object_detection_actual_column_names=object_detection_actual_column_names,
)

response = arize_client.log(
    model_id= "CV-object-detection",
    model_version= "v1",
    model_type=ModelTypes.OBJECT_DETECTION,
    environment=Environments.PRODUCTION,
    dataframe=df,
    schema=schema
)

Google Colaboratory

For more details on Python Batch API Reference, visit here:

Pandas Batch Logging

Object Detection Prediction & Actual Values

Arize supports logging object detection prediction and actual values using the ObjectDetectionColumnNames object, which can be assigned to the prediction/actual schema parameters, object_detection_prediction_column_names and object_detection_actual_column_names.

Object prediction or actual declaration is required to use the object detection model type in Arize.

class ObjectDetectionColumnNames(
    bounding_boxes_coordinates_column_name: str
    categories_column_name: str
    scores_column_name: Optional[str] = None # actual ground truth labels wont have scores
)

Embedding Features

In addition to object detection prediction and actual values, Arize supports logging the embedding features associated with the images in an object detection model using the EmbeddingColumnNames object.

• The vector_column_name should be the name of the column where the embedding vectors are stored. The embedding vector is the dense vector representation of the unstructured input. ⚠️ Note: embedding features are not sparse vectors.
• The link_to_data_column_name should be the name of the column where the URL links to the source images are stored.

{ 
    "embedding_display_name": EmbeddingColumnNames(
        vector_column_name="image_vector", 
        link_to_data_column_name="image_link" 
    ) 
}

See here for more information on embeddings and options for generating them.