✨Arize Copilot

Use AI to build and troubleshoot AI

Arize Copilot is an AI assistant to help AI engineers build and improve their applications. Copilot has support for over 30+ skills including prompt optimizer, eval builder, AI search, etc. It is integrated into the product and also available as a Chat Experience.

AI-Powered Skills

Copilot has 30+ skills across both generative applications and tabular/CV use cases.

Skills for LLM Applications

Skill	Description
AI Search - Column Search	Searches a specific column based on the user’s input to find relevant data. Example: "Find me confused inputs."
AI Search - Table Search	Searches across the entire table to identify patterns, anomalies, or outliers. Example: "Find inputs that reference pricing that are hallucinated."
AI Search - Text to Filter	Generates query filters based on natural language commands. Example: "Filter by input contains SDK."
AI Search - LLM Analysis Lite	Provides suggestions for search results and finds patterns in the data. Example: "What are the top 5 types of questions asked?"
Create Custom Evaluations	Writes a tailored eval for your application based on specified goals or data analysis.
Diagnose RAG Issues	Analyzes responses in the retrieval process, ensuring relevance and accuracy, offering improvements.
Optimize Prompts	Optimizes prompts to enhance response quality or address specific issues.
Summarize Evaluation Metrics	Assesses and summarizes evaluation metrics, providing suggestions for enhancing performance.
ArizeQL Generator	Generates custom metrics by translating natural language descriptions or existing code (e.g., SQL, Python) into AQL for easy application.

Skills for ML & CV Models

Skill	Description
Get Model Insights	Provides a high-level analysis of model performance, including trends over time, prediction volumes, and drift. Best for general inquiries, not suited for detailed debugging.
Cohort Performance Analysis	Analyzes model performance across different cohorts or slices of data, identifying poorly performing segments. Provides insights into behavior over a specific period.
Detect Data Drift	Pinpoints sudden input quality issues by examining features and tags for drift. Compares current distributions to a baseline to detect significant shifts.
Check for Missing Data	Analyzes input data to report the percentage of missing data in features and tags, highlighting any sudden spikes or changes that could impact model inputs.
Assess Feature Data Quality	Assists in debugging issues by analyzing dataset metrics and focusing on specific dimensions. Identifies critical changes and provides actionable suggestions.
Evaluate Distribution Shifts	Analyzes a dimension’s distribution to understand shifts in percentage over time.
Review Cardinality Trends	Analyzes changes in the cardinality of features and tags over time, highlighting unusual variations that may indicate data quality issues.
Embedding Summarization	Provides concise summaries of embedding data, helping you quickly understand patterns and insights from your models' embeddings
ArizeQL Generator	Generates custom metrics by translating natural language descriptions or existing code (e.g., SQL, Python) into AQL for easy application.

Data Privacy

Arize Copilot is built on Azure OpenAI because of its built-in security and compliance features. This ensures that customer data is protected and not directly exposed to third-party providers. Here's how it works:

• Data Processing: Azure acts as the data processor for the prompts and outputs sent to and generated by Copilot. The models are stateless, meaning no prompts or outputs are stored in the model.

• No Data Sharing or Model Improvement:

  • Inputs and outputs of Copilot are NOT used to improve OpenAI models.

  • They are NOT used to improve any Microsoft or third-party products or services.

  • They are NOT used for automatically improving Azure OpenAI models.

• Full Control by Microsoft: The Azure OpenAI Service is fully controlled by Microsoft, and the OpenAI models are hosted in Microsoft’s Azure environment. The service does NOT interact with any other OpenAI-operated services, such as ChatGPT or the OpenAI API.

• Security & Compliance: Azure OpenAI ensures that we meet industry-standard security and compliance measures, protecting your data throughout the process.

For a detailed breakdown of the data flow and additional privacy measures, refer to the diagram below:

To read more about Azure Data privacy, see their documentation here. If you have further questions or need more clarification on how your data is managed, feel free to contact our team at [e-mail].

Third-Party Integrations

Copilot includes a support skill designed to help answer user questions. When you ask a support-related question, the question itself is sent to a third-party service, RunLLM, for processing. It's important to note that:

• Limited Data Sharing: Only the specific question you ask is shared with RunLLM. No additional model information or user data is shared beyond the question itself.

• User Control: You retain control over your interaction with this skill. If at any time you wish to modify or revoke your consent to share support questions with RunLLM, please contact us at [e-mail]

• Disclaimer Acknowledgement: Before using the support skill, users must acknowledge a one-time disclaimer outlining the involvement of RunLLM.