Skip to content

AI Models Troubleshooting

This guide covers common failures when importing, serving, fine-tuning, or exporting local models.

View as Markdown Open the plain-text version of this page.

Gateway returns 503 for a self/<model-id> request#

What it looks like: The Gateway responds with a model-unavailable error even though the model shows as installed.

Common causes:

  • services/python-inference is not running or PROVON_PYTHON_INFERENCE_URL points to the wrong host/port.
  • The model service was started but has not reached the running state yet.
  • The requested endpoint (for example, chat/completions) is not listed in the profile's endpoints.

What to do:

  1. Check the Python inference service health directly:
    bash
    curl "$PROVON_PYTHON_INFERENCE_URL/health"
  2. Check the runtime status to see the service state and the selected profile:
    bash
    curl "$PROVON_API_URL/projects/$PROJECT_ID/models/runtime" \
  -H "Authorization: Bearer $PROVON_API_KEY"
  3. Call the Python service directly with the profile's routeModel to rule out Gateway routing:
    bash
    curl "$PROVON_PYTHON_INFERENCE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $PROVON_PYTHON_INFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<route-model>", "messages": [{"role":"user","content":"hi"}]}'

Runtime reports an engine as unsupported#

What it looks like: provon model install fails with "Engine X is not supported on this machine", or the API status shows supportedOnThisMachine: false.

Common causes:

  • PROVON_PYTHON_INFERENCE_URL is not configured.
  • The Python inference service is reachable but the engine extra is not installed (for example, you are trying to serve a Safetensors model but only installed the llama.cpp extra).

What to do:

  1. Confirm the URL:
    bash
    echo "$PROVON_PYTHON_INFERENCE_URL"
curl "$PROVON_PYTHON_INFERENCE_URL/health"
  2. Reinstall the Python inference service with the required engine extra:
    bash
    pip install -e ".[llama.cpp]"      # for GGUF
pip install -e ".[vllm]"           # for Safetensors via vLLM
pip install -e ".[sglang]"         # for Safetensors via SGLang
pip install -e ".[all]"            # install every engine

Remember that Node and the CLI never install engine binaries themselves; they only report whether the remote Python inference service can serve a given engine family.

Model download is interrupted or stuck#

What it looks like: provon model install or the download API returns a job that stays in running forever, or the CLI exits before completion.

What to do:

  1. Check activity to see the download job state:
    bash
    curl "$PROVON_API_URL/projects/$PROJECT_ID/models/activity" \
  -H "Authorization: Bearer $PROVON_API_KEY"
  2. Partial downloads are stored as .part files in PROVON_MODELS_DIR. If a job is stuck, cancel it and start a new download; the runtime resumes from the existing .part file when possible.
  3. If you are using the CLI, the process must stay alive while the download runs. For long downloads, prefer using the Node server API instead, where downloads continue as long as the server process runs.

Catalog is empty after restarting the Node server or CLI#

What it looks like: provon model list --all returns no models, or the catalog query returns an empty page after a restart.

Why: The catalog cache and runtime state are kept in memory only. When the process restarts, the cache is gone. Imported models that have been downloaded are rediscovered from local manifest.json files, but the upstream catalog browseable list must be refreshed.

What to do:

bash
curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog-refreshes" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Or with the CLI:

bash
provon model search --query llama --limit 20

A checkpoint cannot be deployed#

What it looks like: POST /fine-tuning/checkpoints/:checkpointId/deploy returns checkpoint "..." is not registered for inference or a deployment timeout.

Common causes:

  • The checkpoint does not have a profileId, which means the training job did not produce a runtime profile. Check the job output format and whether the Python fine-tuning service reported success.
  • services/python-inference is not reachable.
  • The checkpoint path is not readable from the Python inference service host.

What to do:

  1. Read the checkpoint to confirm it has a profileId:
    bash
    curl "$PROVON_API_URL/projects/$PROJECT_ID/fine-tuning/checkpoints/$CHECKPOINT_ID" \
  -H "Authorization: Bearer $PROVON_API_KEY"
  2. Confirm the Python inference service is healthy and can access PROVON_MODELS_DIR.
  3. Try starting the model manually with the profile id to get a clearer error:
    bash
    curl -X PUT "$PROVON_API_URL/projects/$PROJECT_ID/models/$PROFILE_ID/profiles/$PROFILE_ID/service" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Fine-tuning job fails immediately#

What it looks like: The job status becomes failed seconds after creation.

Common causes:

  • The Dataset does not use the chat/v1 schema or has zero examples.
  • The base model is not known to the runtime.
  • Node cannot reach services/python-fine-tuning.
  • PROVON_MODEL_FINE_TUNING_WORK_DIR is set differently for Node and the Python service, so the manifest path is invalid from one side.

What to do:

  1. Verify the Dataset schema and example count.
  2. Confirm the base model is imported and, if needed, downloaded.
  3. Check the job error message and the Python fine-tuning service logs.
  4. Ensure Node and the Python service share the same absolute work directory.

GGUF export fails#

What it looks like: An export job with format: "gguf" fails during conversion or quantization.

Common causes:

  • PROVON_LLAMA_CPP_PATH points to a directory that is missing convert_hf_to_gguf.py or llama-quantize.
  • The requested quantMethod is not supported for the source model architecture.

What to do:

  1. Confirm the llama.cpp checkout has the required scripts:
    bash
    ls "$PROVON_LLAMA_CPP_PATH"/convert_hf_to_gguf.py
ls "$PROVON_LLAMA_CPP_PATH"/build/bin/llama-quantize
  2. Use a common quantization such as q4_k_m for your first export.
  3. Read the export job record for the exact error message.

Multiple Node processes share the same data directory#

What it looks like: Downloads restart unexpectedly, service ports conflict, or models flip between running and stopped states.

Why: Only one Node process should own a given PROVON_MODELS_DIR at a time. Runtime state (downloads, service bookkeeping, catalog cache) is in memory, so two processes cannot coordinate.

What to do: Run a single Node server for each PROVON_MODELS_DIR. If you need isolation, use separate PROVON_MODELS_DIR values for different environments.

Getting more help#

If these steps do not resolve the issue:

  1. Collect the runtime status, activity, and the relevant job or service records.
  2. Collect logs from services/python-inference or services/python-fine-tuning.
  3. Open an issue with the request id, model id, profile id, and the exact error response.

For deployment topology questions, see Self-hosting Node.