Guide

Model Providers

Route cloud, local, private, and supply-node models through one Gateway contract.

Provon Model Providers

Model providers are the model backends you route through Provon. They can be cloud providers, local/private providers exposed with the self/ namespace, or external supply-node providers. This is different from self-hosting Provon itself. For deploying Provon on your own infrastructure, see deployment.md.

Provon treats model providers as first-class Gateway targets. They let teams compare local, private, supply-side, and cloud-backed inference behind the same OpenAI-compatible contract, while keeping trace capture, guardrails, routing, cost analysis, Playground, and workflow loops consistent.

Gateway Contract

Runnable local or private provider targets use the self/ prefix.

{
  "model": "self/qwen2.5-0.5b-instruct-q4-k-m"
}

From an agent application's perspective, this still goes through the Provon Gateway. That means local/private provider calls can participate in the same evidence and control loop as cloud provider calls:

Gateway attempts and errors are captured as trace evidence.
Token, cost, and latency signals remain visible in the Workspace.
Guardrails can run on the request/response path.
Playground can compare prompt, model, and parameter variants.
Workflow evaluators can use provider-backed runs as production evidence.

Provider Paths

Provider path	Typical artifacts	Example capabilities
llama.cpp `llama.cpp`	GGUF, quantized and split artifacts	Local chat/completions on CPU or GPU-backed runtimes
MLX MLX LM	Apple Silicon language models	Local text generation through `mlx_lm.server`
MLX Audio MLX Audio	Speech models such as ASR/TTS assets	Local transcription, speech generation, and audio pipelines
ONNX ONNX	Transformers.js-compatible models	Embeddings, rerank, moderation, classification, pipelines
vLLM vLLM	Safetensors snapshots	High-throughput model serving when a vLLM server is available

When To Use Model Providers

Local or private provider targets are useful when you need:

Lower-cost routing for simple or high-volume workloads.
Local experimentation before routing production traffic to a model.
Private inference paths for sensitive workloads.
Direct comparison between cloud and local behavior under the same trace format.
A stable fallback target when external provider behavior or pricing changes.

Typical Workflow

Configure or start a local provider through the Workspace or the Node/Desktop runtime.
Expose the model as a self/* Gateway target.
Route agent requests through the OpenAI-compatible Gateway.
Inspect traces, cost, latency, and output behavior in the Workspace.
Use Playground and workflow evals to compare the provider target with cloud models.
Add routing policies or guardrails once the target is ready for regular traffic.

Deployment Relationship

Model execution is a provider concern. The most direct Provon shapes are:

desktop for single-machine experimentation and local model management.
services/node-server for team or private-cloud deployments that also run local/private providers.
provond for connecting external OpenAI-compatible endpoints as supply nodes.

For Provon deployment shapes and storage choices, see deployment.md.

Current Scope

Provon provides the control plane and Gateway contract around local and private inference providers. It does not try to replace every model server. Engines such as llama.cpp, MLX, ONNX, and vLLM remain the execution layer; Provon makes their behavior observable, comparable, governable, and usable from the same agent-facing API surface.