# Troubleshooting

Start with the first failed verification step. A Gateway response, an ingested trace, a diagnostic
Run, a Finding, and a GitHub handoff are separate stages.

## Provon Does Not Start

Check:

```bash
node --version
pnpm --version
pnpm dev
```

- Node.js must be 22 or newer.
- The local Workbench port defaults to `3000`; the development API uses a separate internal port.
- Another process or Provon instance must not hold the configured DuckDB file lock.
- Relative local storage paths resolve from the repository root.

Use a custom Workbench port when needed:

```bash
pnpm dev -- --port 3100
```

## The Workbench Loads But API Calls Fail

- Read the `pnpm dev` output for the Workbench and API origins.
- Confirm `PROVON_API_ORIGIN` was not overridden to an unavailable server.
- When using a reverse proxy, forward the original host and scheme.
- Confirm the signed-in user belongs to the requested project.

## OTLP Returns `401` Or `403`

Verify the header format:

```bash
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $PROVON_API_KEY"
```

The key must be a project API key with `telemetry:ingest`. Provider API keys and GitHub tokens are
not valid OTLP credentials.

## OTLP Succeeds But No Trace Appears

Check these in order:

1. The exporter uses OTLP/HTTP, not OTLP/gRPC.
2. The endpoint is the Provon origin, or the signal-specific URL ends in `/v1/traces`.
3. The exporter is flushed before the process exits.
4. The trace is in the same project as the API key.
5. The selected Workbench time range includes the span timestamps.

Cloudflare ingestion is asynchronous. A `200` means the payload was durably staged and queued; trace
summaries can appear after the raw write completes.

The background worker parses and validates individual records after the response. If a known trace
never appears, inspect ingest-worker and dead-letter logs for malformed OTLP envelopes or rejected
records.

## OTLP Returns `413`, `422`, `501`, Or `503`

- `413`: reduce the exporter batch size. The default decoded request limit is 10 MiB.
- `422`: use a supported content type and content encoding, and confirm protobuf decoding is
  configured.
- `501`: the deployed runtime does not enable the requested signal.
- `503`: retry with exponential backoff and honor `Retry-After` when queue admission is saturated.

Do not retry authentication or validation errors without changing the request.

See the [OTLP/HTTP API](./otlp-api.md) for the complete acceptance and retry contract.

## Traces Appear Without Conversation Grouping

Add a stable `gen_ai.conversation.id` to the agent spans. Reusing a trace ID is not a substitute for
conversation identity because one conversation may contain multiple traces. `session.id` is retained
as an attribute but is not the canonical Provon conversation key.

Also verify:

- the identifier is attached consistently across turns;
- timestamps and parent span IDs preserve event order;
- user, assistant, and tool content was retained when diagnosis requires it.

Provon also recognizes `conversation.id`, `conversation_id`, `provon.conversation_id`, `thread.id`,
and `thread_id`.

## A Trace Is Visible But Not Diagnosable

Trace visibility only requires valid spans. Diagnostic Rules need a reconstructable trajectory.
Verify that the trace includes:

- a bounded root operation and ordered child spans;
- the user goal in `gen_ai.input.messages`;
- the terminal answer in `gen_ai.output.messages`;
- tool call IDs, arguments, results, failures, and retries;
- error span status plus `error.type`;
- raw evidence allowed by the payload-retention policy.

See [Trace model and instrumentation](./tracing-model.md#diagnosis-readiness).
Use the [diagnosis-ready tracing checklist](./tracing-best-practices.md) to review the complete run.

## Trace Search Misses An Older Trace

Trace list and search default to the last 24 hours. Select a wider Workbench range or pass a larger
API or CLI range such as `--range 7d`.

Content, input, and output search uses projected previews and remains available with projected-only
retention. It can match only the retained previews; the full raw body is unavailable for later
inspection.

## Agent Transcript Sync Finds No Files

- Point `provon sync` at a directory containing `.json` or `.jsonl` transcripts.
- Use `--once --dry-run --print-sample` to inspect detection without uploading.
- Confirm the process can read nested source directories.
- Wait for an actively written JSONL record to finish; the scanner ignores an incomplete final line
  until the file has been quiet long enough.
- Inspect the command's `failures` and `warnings` arrays.

## AI Gateway

Use [Gateway troubleshooting](./gateway-troubleshooting.md) for the request-ordered diagnostic path:

1. verify project API-key authentication and capability;
2. inspect model resolution and project policy;
3. inspect the selected provider attempts;
4. verify Trace Capture and telemetry delivery.

It covers `401`, `402`, `403`, `404`, `409`, `413`, `415`, `429`, and `503` responses, streaming
failures, provider errors, missing traces, and incomplete conversation evidence.

## Auto-Diagnose Produces No Findings

This can be correct. Healthy conversations and weak candidates should not create Findings.

Verify:

1. **Auto-diagnose** is enabled. New projects default to disabled.
2. The relevant Rules are enabled.
3. The conversation has been inactive for at least 10 minutes.
4. A stable conversation identifier and usable trajectory evidence exist.
5. Rule keywords match the captured conversation text.
6. Signal and diagnostic-confidence thresholds are not filtering the candidate.
7. Model adjudication has a usable model when the Rule mode is `auto`.

To separate scheduling from detection, open **Rules** and backfill the last 24 hours. Use
`deterministic-only` when testing without an adjudication model.

## A Rule Run Is Skipped

Common skip reasons:

- the Rule was disabled;
- automatic scheduling was turned off;
- the conversation no longer exists;
- newer activity superseded the queued snapshot;
- no usable trajectory could be reconstructed;
- configured keywords did not match.

Skipped Runs do not publish Findings.

## Create Repair Issue Fails

The current repair handoff requires an enabled GitHub connector.

Check:

- **Connectors > GitHub** is enabled;
- a token or OAuth connection is stored;
- **Default owner** and **Default repo** are set;
- the token can create issues in that repository;
- the deployment has an integration secret for encrypted connector credentials.

For OAuth, configure `GITHUB_INTEGRATION_CLIENT_ID` and
`GITHUB_INTEGRATION_CLIENT_SECRET`. For local testing, expand the connector and provide a token
manually.

## A Finding Is Handed Off But Never Resolves

Provon stores references as `github:owner/repo#number`. Resolution sync needs:

- the GitHub connector to remain enabled;
- permission to read the issue and associated pull requests;
- the background repair-status loop to be running;
- either a merged associated pull request or a closed issue.

Closing a Finding in the Workbench is not a substitute for the external repair outcome.

## Seed Data Fails

`pnpm seed:local` intentionally requires empty metadata and telemetry databases. It will not merge
demo data into an existing workspace.

Stop the local server before seeding so DuckDB is not locked. Use a separate configured database if
you need to preserve existing local data.

## Still Blocked

Capture:

- the failing command or HTTP method and path;
- the response status, error code, and request ID;
- the runtime shape (Node or Cloudflare);
- the relevant project ID;
- whether the previous stage in the loop succeeded.

Do not include project API keys, provider credentials, GitHub tokens, prompts, or private payloads
in a public issue.

## Related Docs

- [Tracing overview](./tracing.md)
- [Tracing quickstart](./tracing-quickstart.md)
- [OpenTelemetry setup](./opentelemetry.md)
- [OTLP/HTTP API](./otlp-api.md)
- [Tracing attribute reference](./tracing-attributes.md)
- [Diagnosis-ready tracing](./tracing-best-practices.md)
- [Trace diagnosis with CLI](./trace-diagnosis.md)
- [Production tracing](./tracing-production.md)