# AI Models Quickstart

Run a local open-weight model behind the Provon Gateway in a few minutes. This guide uses a small
GGUF model so you can follow it on a laptop with CPU inference.

## Prerequisites

- A running Provon Node server or the `provon` CLI.
- A running `services/python-inference` process with the `llama.cpp` engine installed.
- A project API key with `models:manage` and `gateway:invoke` capabilities.

## Start The Python Inference Service

In one terminal, start the inference service:

```bash
cd services/python-inference
pip install -e ".[llama.cpp]"
PROVON_PORT=8000 PROVON_MODELS_DIR=./models python -m provon_inference
```

Verify it is healthy:

```bash
curl http://127.0.0.1:8000/health
```

## Configure The CLI

Point the CLI at the Python inference service and at your Provon workspace:

```bash
export PROVON_PYTHON_INFERENCE_URL=http://127.0.0.1:8000
export PROVON_API_URL=https://api.provon.dev/v1
export PROVON_API_KEY=your-project-api-key
```

If you are running the Node server locally, use `http://127.0.0.1:3000` for `PROVON_API_URL`.

## Import A Model

Search for a small GGUF model:

```bash
provon model search --query "qwen2.5 0.5b gguf" --format gguf --limit 5
```

Import the exact repository:

```bash
provon model import Qwen/Qwen2.5-0.5B-Instruct-GGUF
```

The command prints the selected profile id. Save it for the next step.

## Download And Serve

Install (download) the model artifacts:

```bash
provon model install Qwen/Qwen2.5-0.5B-Instruct-GGUF --profile <profile-id>
```

Then serve it in the foreground:

```bash
provon model serve Qwen/Qwen2.5-0.5B-Instruct-GGUF --profile <profile-id> --foreground
```

The command prints the Gateway model id, typically `self/qwen2.5-0.5b-instruct-gguf`.

## Send A Request Through The Gateway

In another terminal, send a chat completion request:

```bash
curl "$PROVON_API_URL/gateway/v1/chat/completions" \
  -H "Authorization: Bearer $PROVON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "self/qwen2.5-0.5b-instruct-gguf",
    "messages": [{"role": "user", "content": "Say hello"}]
  }'
```

If the Gateway resolves the model to your running profile, the response comes from the local
Python inference service.

## Verify The Trace

Open the Provon Workbench or query the Traces API for the request you just sent. The trace shows:

- the Gateway model id (`self/...`);
- the upstream `routeModel` the runtime selected;
- the response latency and token usage.

This confirms the local model is integrated into the evidence loop.

## Send A Request Directly (Optional)

For debugging, you can also call `services/python-inference` directly using the profile's `routeModel`:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<route-model>",
    "messages": [{"role": "user", "content": "Say hello"}]
  }'
```

Use `provon model get <model-id> --profile <profile-id>` to read the `routeModel`.