Custom OpenAI-compatible endpoints

The Custom (OpenAI-compat) provider accepts any endpoint that implements the OpenAI chat completions API. That covers a lot:

• Self-hosted vLLM with the OpenAI server
• Ollama behind a reverse proxy
• LocalAI, text-generation-webui, LM Studio
• Managed endpoints — Fireworks, Anyscale, Replicate, Modal, your own AWS Bedrock proxy
• Azure OpenAI (with a thin shim for the URL pattern)

What you need

1. A base URL (e.g. https://my-vllm.example.com/v1). Must end where /chat/completions can be appended.
2. An API key (any string — your endpoint decides what to validate).
3. A model name we'll use in requests (e.g. mistralai/Mistral-Large-Instruct).

Request shape

We POST to {baseUrl}/chat/completions with the standard payload:

POST /chat/completions
Authorization: Bearer <your-key>
Content-Type: application/json

{
  "model": "<your-model>",
  "messages": [
    { "role": "system", "content": "..." },
    { "role": "user", "content": "..." }
  ],
  "stream": true,
  "temperature": 0.7,
  "max_tokens": 1200
}

Streaming

We use SSE streaming for drafts (user sees tokens as they generate). If your endpoint doesn't support streaming, uncheck "Supports streaming" on the provider form. We'll fall back to non-streaming.

Auth flavors we support

• Authorization: Bearer <key> (default)
• api-key: <key> header (Azure-style)
• Querystring ?api_key=<key> (Gemini-style, but use the native Google provider for that)

Testing first

Run a curl against your endpoint with the exact payload shape before configuring distro. If curl works, distro will work. If curl fails, distro will too. Most custom-endpoint issues are upstream.