Model Providers

Model providers are the upstream inference targets behind Provon AI Gateway. The project provider catalog describes what an adapter can support; Provider Keys and model mappings de

View as Markdown Open the plain-text version of this page.

Running a self/* model is different from self-hosting the Provon platform. The former selects a local model-runtime target; the latter determines where the Workbench, API, Gateway, evidence, and jobs run.

application
  -> Provon project API key
  -> AI Gateway
  -> project Provider Key
  -> upstream credential and model

Provider Model#

Four layers have distinct responsibilities:

Layer	Responsibility
Provider catalog	Built-in adapter, endpoint families, features, aliases, and model metadata
Provider Key	Project-scoped target, secret, priority, timeout, endpoint, and headers
Model mapping	Public Gateway model ID to upstream model ID plus endpoint/features
Pricing definition	Cost estimation and trace enrichment only

Pricing is not routing truth. Adding a pricing definition does not make a model routable.

Choose Target Access#

Provon can reach models through three credential and billing boundaries:

Access mode	Credential owner	Billing boundary	Availability
Bring your own key (BYOK)	Project Provider Key	Upstream provider account	Node and hosted deployments
Provon Cloud managed model	Provon Cloud	Organization AI Gateway credit balance	Deployments that expose PTB targets
Self-hosted model	Model runtime	Deployment compute	Node runtime

BYOK is the portable default. The application authenticates to Provon with its project API key; Provon authenticates upstream with the Provider Key.

Managed model requests reserve estimated cost from Cloud AI Gateway credits before execution and settle against actual usage afterward. Insufficient balance returns 402 with INSUFFICIENT_GATEWAY_CREDITS. A self-hosted deployment does not gain managed provider access by adding a pricing definition. The runtime must expose a PTB upstream resolver; default BYOK-only Gateway surfaces cannot route managed Provon Cloud models.

Self-hosted models use the self/ namespace and do not need an upstream API credential.

Provider Types#

Type	Examples	Configuration
Native provider adapter	Anthropic, Gemini, Cohere, Workers AI	Provon translates supported OpenAI shapes
OpenAI-compatible provider	OpenAI, OpenRouter, Groq, Mistral, DeepSeek	Provon forwards compatible endpoint shapes
Deployment-specific provider	Azure OpenAI, Gemini Enterprise	Requires a project or deployment base URL
Custom public endpoint	OpenAI-compatible vendor endpoint, NVIDIA NIM	Requires an allowed public HTTP(S) base URL
Model runtime	llama.cpp, MLX, vLLM, python-transformers and other supported engines	Uses the `self/` namespace and Node runtime

The runtime provider catalog is the source of truth. Query it instead of maintaining a static provider list in application code:

export PROVON_API_URL="https://api.provon.dev/v1"

curl "$PROVON_API_URL/gateway/providers" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Connect A Provider#

In the Workbench:

Open Providers and choose a provider.
Under Provider keys, create a key with a descriptive name.
Add the upstream API key.
Keep the official endpoint, or enter an allowed override where the provider supports one.
Choose All or Specific model mapping mode.
Keep the Provider Key enabled.
Send a test request and verify its trace.

Applications never receive the upstream credential. They call Provon with a project API key that has gateway:invoke.

For a managed Provon Cloud model, inspect the model in Providers and fund the organization under Billing -> Cloud AI Gateway credits. For a model-runtime target, install and start the model under the self-hosted model surface instead of creating a cloud Provider Key.

Self-Hosted Models#

A self-hosted model runs inside your own infrastructure and is exposed as a self/<model-id> Gateway target. Node manages the model catalog, engine installation, artifact downloads, and service lifecycle, while the actual inference runs in services/python-inference.

To run a self-hosted model:

Import the model from Hugging Face Hub or another supported catalog.
Install the required inference engine.
Download the model profile you want to serve.
Start the model service.
Route Gateway requests to self/<model-id>.

See Self-hosted model runtime for the complete workflow and API. See Model fine-tuning for training project-specific checkpoints and deploying them behind the Gateway.

Provider Key Priority#

A project can configure multiple Provider Keys for the same provider. Drag keys to set priority. Matching requests use the first eligible key after capability, policy, health, and load-balancing decisions.

Use separate Provider Keys for:

regions or provider deployments;
independent upstream accounts;
isolated model allowlists;
distinct timeout or header requirements;
ordered recovery targets.

Do not duplicate a key only to rename the same secret. Use one target per operational boundary.

Model Mappings#

Provider Keys support two modes:

Mode	Behavior
All	The key can route provider-compatible model IDs without explicit mappings
Specific	Only enabled model mappings are eligible

An explicit mapping contains:

Field	Meaning
AI Gateway model	Public model ID sent by the application
Upstream model	Model ID sent to the provider
Endpoint types	Endpoint families this mapping may serve
Features	Streaming, tool calling, vision input, reasoning, or other requirements
Context window	Selection and UI metadata
Max output tokens	Selection and UI metadata
Pricing profile	Cost calculation metadata

Example:

AI Gateway model: support-model
Upstream model:   gpt-5-mini-2026-06-01
Endpoints:        chat-completions, responses
Features:         streaming, tool-calling, structured-outputs

The application can then send the provider-qualified public ID:

{
  "model": "openai/support-model"
}

Plain "support-model" requests require a matching Model Policy or registry inference. See Routing and reliability.

Capability Discovery#

Do not assume endpoint parity between providers or between two Provider Keys for the same provider.

curl "$PROVON_API_URL/gateway/capability-matrix?includeDisabled=false" \
  -H "Authorization: Bearer $PROVON_API_KEY"

The matrix includes:

recognized endpoint families;
provider-native, translated, bridged, and unsupported status;
provider feature declarations;
enabled Provider Keys and credentials;
model mappings and their endpoint/features.

Filter model selection for the exact request shape:

curl \
  "$PROVON_API_URL/gateway/model-selection?endpoint=responses&features=streaming,tool-calling" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Custom OpenAI-Compatible Targets#

Use the openai-compatible provider for a user-owned endpoint that implements supported OpenAI request and response shapes.

Configure:

a public HTTP(S) base URL; use HTTPS for production traffic;
a Provider Key secret;
explicit model mappings when the target should expose only named logical models;
accurate endpoint and feature declarations for mapped models;
an attempt timeout.

Provon rejects credentials in URLs, query strings or fragments in base URLs, internal hostnames, and private or reserved IP literals. Deployments can inject DNS resolution policy to also reject public names that resolve to any non-public address. Official provider adapters keep their official endpoint fixed unless their manifest explicitly allows an override.

Use self/ models for model-runtime or private-network inference. Do not point a public custom Provider Key at a loopback or internal service.

Runtime And Private Models#

Runnable models use the self/ namespace:

{
  "model": "self/qwen3.5-2b"
}

The local model runtime owns artifact storage, persisted lifecycle state, inference health checks, and Gateway target resolution. The logical ID after self/ is stable across runtime profiles; the Gateway selects a healthy compatible profile and sends that profile's exact route ID to the standalone Python inference service. A model is routable only when:

its model artifact is installed;
the required engine is available;
the service is running and healthy;
the model exposes the requested endpoint.

Import an exact Hugging Face repository when it is not present in the synchronized catalog:

provon model import mlx-community/gemma-4-e2b-it-4bit

Imported repositories remain pinned in the local catalog across catalog refreshes.

The model engine remains responsible for inference. Provon supplies the Gateway contract, routing, evidence capture, guardrails, and comparison surface.

Cloudflare deployments do not bundle the local model runtime adapter.

Advanced Provider Key Settings#

Provider Keys can define:

per-attempt timeout;
header overrides sent only to that target;
retry policy;
fallback status and network-error policy;
circuit-breaker policy;
priority and credential ordering.

The current Workbench exposes the commonly used timeout and header controls. Provider Key management APIs also carry the reliability policy fields used by runtime adapters.

Recovery remains capability-aware: an alternate key or model that cannot satisfy the request is not a valid fallback.

Trace Evidence#

When Trace Capture selects a request, the trace can show:

selected provider, Provider Key, and credential fingerprint;
requested, Gateway, and upstream model IDs;
translated or bridged operation shape;
every provider attempt and fallback index;
latency and time to first chunk;
token, cache, reasoning, and cost fields;
normalized provider errors and guardrail outcomes.

See Gateway evidence.

Security#

Store upstream credentials only in project-scoped Provider Keys.
Give calling applications Provon project API keys.
Use Specific mappings for strict model allowlists.
Rotate upstream and project keys independently.
Restrict custom endpoints to intended public hosts.
Treat header overrides as secrets-capable configuration.
Do not expose model servers directly when Gateway is the control point.