# Self-hosted Models

Provon can serve local and private models behind the same OpenAI-compatible Gateway used for cloud
providers. The Node runtime owns model catalog, artifact download, and service lifecycle; the actual
inference runs in `services/python-inference`, a standalone Python service that Node controls
remotely over HTTP.

This is different from [self-hosting the Provon platform](../self-hosting/index.md). A model-runtime target
is one upstream choice inside the Gateway, while self-hosting determines where the Workbench, API,
evidence, and jobs run.

## When To Use It

Use the self-hosted model runtime when:

- prompts or models must stay inside your network;
- you want to run open-weights models on your own GPU/CPU hardware;
- you are fine-tuning models and want to deploy the resulting checkpoints behind the Gateway;
- a cloud provider does not expose the model you need.

Use a cloud Provider Key when operational simplicity, elastic scale, or managed model access matter
more.

## Architecture

```mermaid
flowchart LR
  app["Agent application"] --> gateway["AI Gateway"]
  gateway --> self["self/<model-id>"]
  self --> node["Node runtime"]
  node --> catalog["Model catalog"]
  node --> engines["Engine installs"]
  node --> downloads["Artifact downloads"]
  node --> python["services/python-inference"]
  python --> gpu["GPU / CPU inference"]
```

The Node runtime:

- discovers models through the canonical `@provon/model-catalog` package;
- downloads model artifacts and tracks progress;
- starts, stops, and health-checks the inference service profile by calling `services/python-inference` over HTTP;
- exposes each logical model as a stable `self/<gateway-model-id>` Gateway route.

Node does **not** install inference engines and does **not** spawn Python child processes. Engine execution lives in `services/python-inference`, which you deploy and manage separately. Node only reports whether the remote Python inference service is reachable and which engine families it can route to.

## Supported Engines

| Engine    | Formats typically served | How it runs                        |
| --------- | ------------------------ | ---------------------------------- |
| llama.cpp | GGUF                     | inside `services/python-inference` |
| vLLM      | Safetensors              | inside `services/python-inference` |
| SGLang    | Safetensors              | inside `services/python-inference` |

Engine availability depends entirely on the remote `services/python-inference` process. The runtime reports `supportedOnThisMachine` based on whether `PROVON_PYTHON_INFERENCE_URL` is configured and reachable, and the `installMode` is always `external`. The Workbench uses this status to show which model profiles can be served.

## Import A Model

Models are imported from a catalog source. The primary source is Hugging Face Hub:

```bash
curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/imports/huggingface" \
  -H "Authorization: Bearer $PROVON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repoId": "organization/model-name"}'
```

Importing creates a runtime record and selects downloadable profiles. The model is not downloaded
automatically; start a download for the profile you want to serve.

## Start And Serve A Model

After import, start the download:

```bash
curl -X POST \
  "$PROVON_API_URL/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID/downloads" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

Then start the inference service:

```bash
curl -X PUT \
  "$PROVON_API_URL/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID/service" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

Model identity is split by responsibility:

| ID                          | Purpose                                         |
| --------------------------- | ----------------------------------------------- |
| `catalog.id`                | Internal catalog record                         |
| `runtimeProfile.id`         | Exact install and lifecycle target              |
| `catalog.gatewayModelId`    | Stable model ID used by applications            |
| `runtimeProfile.routeModel` | Exact model ID exposed by the inference service |

Once a compatible profile reports `running`, route Gateway requests to the logical model ID:

```bash
curl "$PROVON_API_URL/gateway/v1/chat/completions" \
  -H "Authorization: Bearer $PROVON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "self/qwen3.5-2b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

The Gateway resolves `catalog.gatewayModelId` to a healthy running profile that supports the
requested endpoint, then rewrites the upstream request to that profile's `routeModel`. Applications
therefore do not need to include engine, format, quantization, or profile details in the model ID.

## Direct Inference Service Requests

For debugging, you can send requests directly to `services/python-inference` using the profile's `routeModel`. This bypasses Gateway routing and is useful when a model reports `running` but Gateway requests fail:

```bash
curl "$PROVON_PYTHON_INFERENCE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $PROVON_PYTHON_INFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<runtimeProfile.routeModel>",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

Use `GET /v1/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID` to read the exact `routeModel` for a profile.

## Runtime Status And Activity

Get the runtime overview:

```bash
curl "$PROVON_API_URL/projects/$PROJECT_ID/models/runtime" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

List active downloads and engine installs:

```bash
curl "$PROVON_API_URL/projects/$PROJECT_ID/models/activity" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

The response includes engine capabilities, installed versions, download progress, and service health.

## Catalog Queries

List models that the runtime can import:

```bash
curl "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog?query=llama&limit=20" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

Query parameters follow the canonical catalog format: `query`, `pipelineTag`, `format`, `engine`,
`limit`, and cursor fields. Refresh the catalog from upstream when needed:

```bash
curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog-refreshes" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

## Engine Availability

Node does not install engines itself. Instead, configure and run `services/python-inference` on a host that has the engine and its dependencies installed. The runtime then reports the engine as available.

If you are using the CLI, `provon model install --engine auto` records that the engine is provided by the remote Python inference service and proceeds to download the model artifacts. It does **not** download or compile llama.cpp, vLLM, or SGLang on your local machine. Use `--engine none` when you want to skip even that bookkeeping step.

In the API, the engine installation endpoint exists for compatibility and returns a `complete` job immediately:

```bash
curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/engines/llama.cpp/installations" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

The meaningful requirement is that `services/python-inference` is reachable and has the engine extras installed. See the [Model CLI](../cli/models.md) and [Self-hosting Node](../self-hosting/node.md) guides for deployment options.

## Configuration

Node runtime storage variables:

| Variable            | Default                    | Purpose                                               |
| ------------------- | -------------------------- | ----------------------------------------------------- |
| `PROVON_BLOBS_DIR`  | `.provon/blobs`            | Staging data; resolves `data/` and `models/` defaults |
| `PROVON_MODELS_DIR` | Sibling `models` directory | Model weights and artifacts                           |

Connection to the Python inference service:

| Variable                          | Default                 | Purpose                                 |
| --------------------------------- | ----------------------- | --------------------------------------- |
| `PROVON_PYTHON_INFERENCE_URL`     | `http://127.0.0.1:8000` | Base URL of `services/python-inference` |
| `PROVON_PYTHON_INFERENCE_API_KEY` | none                    | Bearer token shared with Node           |

Runtime process state (active downloads, running services, engine install bookkeeping, and the catalog cache) is kept in memory only and is reset when the process restarts. Installed models are rediscovered from `<model-dir>/manifest.json` files, and partial downloads can resume from `.part` files. The catalog cache is also in memory; refresh it from upstream after a restart if you need to browse importable models.

The Python inference service is configured separately:

| Variable            | Default    | Purpose                                   |
| ------------------- | ---------- | ----------------------------------------- |
| `PROVON_HOST`       | `0.0.0.0`  | Bind host                                 |
| `PROVON_PORT`       | `8000`     | Bind port                                 |
| `PROVON_API_KEY`    | none       | Bearer token shared with Node             |
| `PROVON_MODELS_DIR` | `./models` | Root directory for cached model artifacts |
| `PROVON_LOG_LEVEL`  | `info`     | Service log level                         |

Engine-specific Python options:

| Variable                                | Default | Purpose                                  |
| --------------------------------------- | ------- | ---------------------------------------- |
| `PROVON_TRANSFORMERS_DEVICE`            | `auto`  | Transformers device override             |
| `PROVON_TRANSFORMERS_DTYPE`             | `auto`  | Transformers dtype override              |
| `PROVON_TRANSFORMERS_TRUST_REMOTE_CODE` | `false` | Allow remote code in Transformers models |

Point Node at the Python inference service through `PROVON_PYTHON_INFERENCE_URL`. Both services must be able to read and write `PROVON_MODELS_DIR` when they run on the same host; when they run on different hosts, configure `PROVON_MODELS_DIR` on the Python service and ensure the paths are consistent from each process's perspective.

## Capability Requirements

The project API key needs:

- `workspace:read` to inspect runtime status, catalog, and activity;
- `models:manage` to import models, start downloads, and control services (the engine installation endpoint exists for compatibility but does not install software);
- `gateway:invoke` to send inference requests through the Gateway.

## Limitations

- Self-hosted model runtime is only available in the Node runtime. Cloudflare Workers do not provide
  local model execution.
- One Node process should own the configured data directory. Multiple active Node
  processes sharing the directory can conflict on downloads and service ports.
- Managed Provon Cloud models and self-hosted models are separate provider paths; adding a pricing
  definition does not make a local model routable.