Self-hosted Models

This is different from self-hosting the Provon platform. A model-runtime target is one upstream choice inside the Gateway, while self-hosting determines where the Workbench, API, evidence, and jobs run.

When To Use It#

Use the self-hosted model runtime when:

• prompts or models must stay inside your network;
• you want to run open-weights models on your own GPU/CPU hardware;
• you are fine-tuning models and want to deploy the resulting checkpoints behind the Gateway;
• a cloud provider does not expose the model you need.

Use a cloud Provider Key when operational simplicity, elastic scale, or managed model access matter more.

Architecture#

flowchart LR
  app["Agent application"] --> gateway["AI Gateway"]
  gateway --> self["self/<model-id>"]
  self --> node["Node runtime"]
  node --> catalog["Model catalog"]
  node --> engines["Engine installs"]
  node --> downloads["Artifact downloads"]
  node --> python["services/python-inference"]
  python --> gpu["GPU / CPU inference"]

The Node runtime:

• discovers models through the canonical @provon/model-catalog package;
• downloads model artifacts and tracks progress;
• starts, stops, and health-checks the inference service profile by calling services/python-inference over HTTP;
• exposes each logical model as a stable self/<gateway-model-id> Gateway route.

Node does not install inference engines and does not spawn Python child processes. Engine execution lives in services/python-inference, which you deploy and manage separately. Node only reports whether the remote Python inference service is reachable and which engine families it can route to.

Supported Engines#

Engine availability depends entirely on the remote services/python-inference process. The runtime reports supportedOnThisMachine based on whether PROVON_PYTHON_INFERENCE_URL is configured and reachable, and the installMode is always external. The Workbench uses this status to show which model profiles can be served.

Import A Model#

Models are imported from a catalog source. The primary source is Hugging Face Hub:

curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/imports/huggingface" \
  -H "Authorization: Bearer $PROVON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repoId": "organization/model-name"}'

Importing creates a runtime record and selects downloadable profiles. The model is not downloaded automatically; start a download for the profile you want to serve.

Start And Serve A Model#

After import, start the download:

curl -X POST \
  "$PROVON_API_URL/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID/downloads" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Then start the inference service:

curl -X PUT \
  "$PROVON_API_URL/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID/service" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Model identity is split by responsibility:

Once a compatible profile reports running, route Gateway requests to the logical model ID:

curl "$PROVON_API_URL/gateway/v1/chat/completions" \
  -H "Authorization: Bearer $PROVON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "self/qwen3.5-2b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

The Gateway resolves catalog.gatewayModelId to a healthy running profile that supports the requested endpoint, then rewrites the upstream request to that profile's routeModel. Applications therefore do not need to include engine, format, quantization, or profile details in the model ID.

Direct Inference Service Requests#

For debugging, you can send requests directly to services/python-inference using the profile's routeModel. This bypasses Gateway routing and is useful when a model reports running but Gateway requests fail:

curl "$PROVON_PYTHON_INFERENCE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $PROVON_PYTHON_INFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<runtimeProfile.routeModel>",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Use GET /v1/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID to read the exact routeModel for a profile.

Runtime Status And Activity#

Get the runtime overview:

curl "$PROVON_API_URL/projects/$PROJECT_ID/models/runtime" \
  -H "Authorization: Bearer $PROVON_API_KEY"

List active downloads and engine installs:

curl "$PROVON_API_URL/projects/$PROJECT_ID/models/activity" \
  -H "Authorization: Bearer $PROVON_API_KEY"

The response includes engine capabilities, installed versions, download progress, and service health.

Catalog Queries#

List models that the runtime can import:

curl "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog?query=llama&limit=20" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Query parameters follow the canonical catalog format: query, pipelineTag, format, engine, limit, and cursor fields. Refresh the catalog from upstream when needed:

curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog-refreshes" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Engine Availability#

Node does not install engines itself. Instead, configure and run services/python-inference on a host that has the engine and its dependencies installed. The runtime then reports the engine as available.

If you are using the CLI, provon model install --engine auto records that the engine is provided by the remote Python inference service and proceeds to download the model artifacts. It does not download or compile llama.cpp, vLLM, or SGLang on your local machine. Use --engine none when you want to skip even that bookkeeping step.

In the API, the engine installation endpoint exists for compatibility and returns a complete job immediately:

curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/engines/llama.cpp/installations" \
  -H "Authorization: Bearer $PROVON_API_KEY"

The meaningful requirement is that services/python-inference is reachable and has the engine extras installed. See the Model CLI and Self-hosting Node guides for deployment options.

Configuration#

Node runtime storage variables:

Connection to the Python inference service:

Runtime process state (active downloads, running services, engine install bookkeeping, and the catalog cache) is kept in memory only and is reset when the process restarts. Installed models are rediscovered from <model-dir>/manifest.json files, and partial downloads can resume from .part files. The catalog cache is also in memory; refresh it from upstream after a restart if you need to browse importable models.

The Python inference service is configured separately:

Engine-specific Python options:

Point Node at the Python inference service through PROVON_PYTHON_INFERENCE_URL. Both services must be able to read and write PROVON_MODELS_DIR when they run on the same host; when they run on different hosts, configure PROVON_MODELS_DIR on the Python service and ensure the paths are consistent from each process's perspective.

Capability Requirements#

The project API key needs:

• workspace:read to inspect runtime status, catalog, and activity;
• models:manage to import models, start downloads, and control services (the engine installation endpoint exists for compatibility but does not install software);
• gateway:invoke to send inference requests through the Gateway.

Limitations#

• Self-hosted model runtime is only available in the Node runtime. Cloudflare Workers do not provide local model execution.
• One Node process should own the configured data directory. Multiple active Node processes sharing the directory can conflict on downloads and service ports.
• Managed Provon Cloud models and self-hosted models are separate provider paths; adding a pricing definition does not make a local model routable.