Skip to content

Docs

Everything you need to compare open inference providers with OpenInference. It takes about a minute to get a first result.

Floating dark code editor windows with softly glowing lines

Quick start

  1. Open the Playground. It starts with Pollinations (free, no key) and Groq.
  2. Type a prompt, or click one of the example prompts.
  3. Press Run (or ⌘ Enter / Ctrl Enter). Every panel receives the same request at the same time and streams its answer.
  4. Use Compare another provider to add a third panel, or the provider menu on each panel to switch.

API keys

Apart from Pollinations, providers need an API key from your own account. Most have a free tier. Click API keys in the playground, and paste a key into the matching field; it saves as you type. Each field links to the page where that provider issues keys.

Keys are saved in your browser's localStorage under openinference:keys. Clear a field to remove that key.

Reading the metrics

TTFT (time to first token)
Time from pressing Run until the first streamed token arrives in your browser, including network time to our relay and to the provider. For reasoning models, the first reasoning token counts.
Total
Time until the stream finished.
tok/s
Output tokens divided by generation time (Total minus TTFT). Uses the token count the provider reports. If a provider does not report usage, it shows ≈ tok/s, estimated at about four characters per token.

Numbers vary run to run with provider load and your connection. Run the same prompt a few times before drawing conclusions.

Parameters

Temperature (0–2) controls randomness; lower is more deterministic. Max tokens (16–4096) caps the length of each answer. Both apply to every panel. The System prompt is optional and is sent as the first message.

The model field accepts any model ID. Suggestions appear as you type, but providers change their catalogs often; if a model is retired, the panel shows the provider's error message.

Sharing & cURL

The URL keeps your provider selection, e.g. /playground?p=groq,together,cerebras, so you can bookmark or share a comparison setup. Prompts and keys are never put in the URL.

The terminal icon on each panel copies an equivalent curl command that calls the provider directly. It uses an environment variable such as $GROQ_API_KEY instead of your key:

curl https://api.groq.com/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  -d '{"model": "llama-3.3-70b-versatile",
       "messages": [{"role": "user", "content": "Hello"}]}'

Privacy

  • Your keys live in your browser. They are sent with each request in a header and forwarded to that provider only.
  • Our relay does not store or log prompts, outputs or keys.
  • The provider you call receives your prompt and handles it under its own terms. Don't send anything you wouldn't send to that provider directly.
  • Requests without your own key are rate limited per visitor to keep the free option available.

Adding a provider

Any service with an OpenAI-compatible /chat/completions endpoint can be added. Append an entry to PROVIDERS in lib/providers.ts:

{
  id: "acme",
  name: "Acme Inference",
  tagline: "Short one-liner",
  description: "What makes it different.",
  baseUrl: "https://api.acme.ai/v1",
  envKey: "ACME_API_KEY",      // optional shared server key
  streamUsage: true,           // accepts stream_options.include_usage
  models: ["acme/llama-3.3-70b"],
  website: "https://acme.ai",
  keysUrl: "https://acme.ai/keys",
  docsUrl: "https://docs.acme.ai",
  highlights: ["Free tier"],
}

The relay only calls base URLs from this list. If envKey is set on the server, visitors can use the provider without their own key (subject to the rate limit).

FAQ