Custom LLM Models

Arize supports any model which has an OpenAI-compatible api.

Add your custom model endpoints to begin accessing your model in Arize's prompt playground. Arize makes use of the openai-client to make calls to these endpoints.

Guide

Navigate to the integration page and add an endpoint

Select Custom Model Endpoint

Fill in the details of your endpoint

• Name: the name you wish to give your endpoint - this is the name it will appear as in other areas of the UI such as in the prompt playground

• Model Name: the exact name of the provider's model

• Base Url: the url of your custom endpoint - Arize leverages the openai-client to make calls to your endpoint, so the relevant endpoints are automatically added for chat (/chat/completions) and completions(/completions) when used

• API Key: key used to access your openai-compatible endpoint

• Headers: optional headers to add in your request

Add your endpoint name, model name, base url, api key, and headers

You can now access your endpoint in Arize's prompt playground

You can navigate to the prompt playground by choosing a generative llm model which has prompts and responses and selecting a table row on the performance tracing tab, or by selecting prompt playground on an llm span

Choose custom provider and the endpoint you've specified, and toggle on the Use Chat switch to use the chat endpoint and off to use the legacy completions endpoint