Skip to content

Tracing

Provon tracing preserves the execution evidence needed to understand an agent run, diagnose repeatable problems, and hand repair-ready context to coding agents. A useful trace connects the user goal, model calls, tool execution, retries, errors, participants, and the terminal answer in causal order.

View as Markdown Open the plain-text version of this page.
text
agent activity -> traces -> conversations -> diagnosis -> Findings

A trace records what happened. A Finding is a supported conclusion produced from one or more traces. Healthy conversations with weak or missing evidence should not produce Findings. Capture, investigation, diagnosis, and repair are separate stages: a recorded trace does not by itself prove an issue, and a closed repair item does not by itself prove the issue is gone.

Evidence prerequisites#

A trace view or a Finding only has meaning after the prerequisites below are met. A missing prerequisite shows up as an empty or ungrouped view — not as evidence that the system is healthy.

Prerequisite What it enables Where to check
Project API key with telemetry:ingest OTLP and CLI uploads authenticate; without it intake returns 401/403 API keys in the Workbench; provon auth token set
Reachable intake target Exporter must point at the Provon OTLP endpoint, not a local debugger or a third-party collector OpenTelemetry; Quickstart
Real Gateway connection for guardrail facts Guardrail outcomes, blocked requests, and provider error categories exist only on requests actually proxied through the AI Gateway AI Gateway quickstart
Conversation identity Requests share one Conversation when they carry x-otel-gen-ai-conversation-id (or an alias); CLI transcripts derive the group from their session id Trace model
Rules enabled and a settled conversation Automatic Findings need at least one enabled Rule and a conversation idle long enough to be diagnosed Diagnostic rules

Guardrail facts come from Gateway-side decisions on real upstream calls. A locally generated transcript, a successful mock response, or a closed issue does not prove a guardrail fired, a fix works, or that a regression has been repaired.

Capture paths#

Four user-facing paths put evidence into Provon. They can run side by side; each incoming record keeps its own trace identity and never rewrites another source's id.

OTLP/HTTP proxied requests session events backfill files Instrumented app Provon intake AI Gateway Agent live hooks Transcript sync Traces, Sessions, Findings
Path Typical user Prerequisites Docs
AI Gateway proxy Any app calling models through the Gateway Gateway base URL and key; capture mode enabled Multi-agent tracing
OTLP/HTTP exporter Own service, SDK, or Collector Exporter endpoint plus a telemetry:ingest key OpenTelemetry
Live agent hooks Claude Code, Codex, Cursor during a session provon ingest hooks install; current live status listed here Quickstart
Transcript backfill Claude Code, Codex, Pi persisted sessions Local transcript directory; authenticated CLI Agent transcripts

Hierarchy#

User Conversation Trace A Trace B Root span Tool span Events and attributes
Unit Built from Purpose
Conversation x-otel-gen-ai-conversation-id on Gateway requests; transcript session id on CLI paths One working session or agent thread; groups traces for review
Trace One incoming request batch (Gateway) or one session file/run (CLI) Ordered work unit: root span plus child spans
Span A request, session root, tool call, or step Timing, model, tool name, usage, errors
Event / attribute Messages, tool results, token usage, status codes Evidence payload attached to a span

The Workbench presents this hierarchy under the Traces navigation section: the Traces list, the Sessions (conversation) console, and the Users list, with Tree and Trajectory views on a detail page. See Explore traces.

Capture semantics at a glance#

Question Answer
What does 200 on intake mean? Payload decoded, authorized, and accepted into processing. It is not proof the trace is rendered yet; read after processing, not immediately after the POST
Which signals are accepted? Traces (and logs where the runtime enables them); unsupported signals receive 501; code and retry behavior are listed in OpenTelemetry
Payload size Default decoded OTLP request limit is 10 MiB; a single sync upload batch defaults to 8 MiB; oversized payloads receive 413
Duplicate uploads Stable transcript-derived ids and per-file content-hash state keep sync idempotent; retries do not duplicate traces
Secrets on transcript sync The CLI redacts sensitive key names and known secret patterns before upload; see Agent transcripts
What stays local Transcript files themselves; sync state (content hashes, upload timestamps) in the CLI state directory

Diagnosis and repair checks#

  1. Evidence exists — at least one trace appears under Traces, and under Sessions when conversation grouping matters.
  2. Context is sufficient — model, tool calls, timings, and grouping are visible. Thin traces (metadata only, no message or tool payloads) rarely satisfy a Rule's evidence floor.
  3. Rules are enabled — the Findings page opens the Diagnostic rules popover; disabled Rules produce no Findings.
  4. Idle window passed — automatic diagnosis runs after the inactivity delay (default 10 minutes); provon traces summarize summarizes but does not diagnose.
  5. Review is a decision, not proof — model review proposes a queue item; a confirmed Finding remains a human or tool judgment, and a healthy conversation can legitimately yield no Finding.
  6. Repair closes only when evidence agrees — see repair semantics. Closing an issue without verification leaves the Finding in awaiting_verification; a fresh recurrence flips it back to regressed.

Documentation path#

User task Start here
Send the first trace and inspect it Tracing quickstart
Configure an SDK, Collector, or exporter OpenTelemetry
Trace Claude Code sessions Claude Code
Trace Codex sessions Codex
Correlate multiple agents in one conversation Multi-agent tracing
Backfill transcript files Agent transcripts
Read traces in the Workbench and CLI Explore traces
Understand spans, ids, and causality Trace model
Keep evidence diagnosis-ready Diagnosis-ready tracing
Run intake safely under real traffic Production tracing
Understand ingestion boundaries and acceptance rules Ingestion boundaries
Go from signals to published Findings Findings overview