# Tracing

Provon tracing preserves the execution evidence needed to understand an agent run, diagnose
repeatable failures, and hand repair-ready context to coding agents. It is not only an LLM request
log: a useful trace connects the user goal, model calls, retrieval, tool execution, retries, errors,
participants, and terminal answer in causal order.

```text
agent activity -> traces -> conversations -> diagnostic Runs -> Findings
```

A trace records what happened. A [Finding](./findings.md) is a supported conclusion produced from
one or more traces. Healthy traces and weak evidence should not produce Findings. Trace capture,
investigation, diagnosis, and Finding persistence are separate product stages.

## Start Here

| Goal                                         | Read                                                                |
| -------------------------------------------- | ------------------------------------------------------------------- |
| Send and diagnose one complete trace         | [Tracing quickstart](./tracing-quickstart.md)                       |
| Add an OpenTelemetry SDK or Collector        | [OpenTelemetry setup](./opentelemetry.md)                           |
| Route the first model call through Provon    | [Gateway quickstart](./gateway-quickstart.md)                       |
| Import a persisted coding-agent session      | [Agent transcript sync](./agent-transcripts.md)                     |
| Design a useful trace before instrumenting   | [Diagnosis-ready tracing](./tracing-best-practices.md)              |
| Look up exact GenAI fields                   | [Tracing attribute reference](./tracing-attributes.md)              |
| Model several agents or services             | [Multi-agent and distributed tracing](./tracing-multi-agent.md)     |
| Investigate in the Workbench                 | [Explore traces](./trace-explorer.md)                               |
| Query or export traces programmatically      | [Tracing CLI](./tracing-cli.md) or [Trace read API](./trace-api.md) |
| Diagnose a trace source without side effects | [Trace diagnosis with CLI](./trace-diagnosis.md)                    |
| Prepare for production traffic               | [Production tracing](./tracing-production.md)                       |

## Choose A Capture Path

| Your application                       | Primary path                                | What Provon captures                                                                                       |
| -------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Calls an OpenAI-compatible model API   | [Gateway](./gateway.md)                     | Model request and response, provider attempts, routing, tokens, cost, latency, errors, and guardrails      |
| Already emits OpenTelemetry            | [OpenTelemetry](./opentelemetry.md)         | Application-defined agent, model, retrieval, tool, and infrastructure spans over OTLP/HTTP                 |
| Uses Claude Code, Codex, or Pi locally | [Agent transcripts](./agent-transcripts.md) | Persisted coding-agent messages, tool calls, results, usage, and source metadata projected into OTel spans |

Use one primary capture path for each model call. Sending an application span and a separate
Gateway record for the same operation can produce duplicate evidence unless the application
deliberately models them as different operations.

The paths can coexist in one project. For example, production agents can use OTLP while persisted
coding-agent sessions arrive through transcript sync.

Provon does not require a proprietary tracing SDK. Direct instrumentation uses OpenTelemetry SDKs,
framework instrumentors, or a Collector that can export OTLP/HTTP. Compatibility means the emitted
telemetry follows the [OTLP transport contract](./otlp-api.md); diagnosis quality still depends on
the trace structure and GenAI evidence.

## What Each Path Can See

Gateway capture is the shortest integration path, but it observes the model boundary rather than
the complete application. It cannot infer a tool result, retrieval decision, or application-side
verification step that never crosses the Gateway.

OTLP instrumentation gives the application control over the complete trace tree. Use it when
diagnosis depends on:

- agent orchestration and handoffs;
- retrieval queries and returned evidence;
- tool arguments, results, failures, and retries;
- application validation or verification steps;
- work performed before or after a model request.

Transcript sync is designed for persisted coding-agent history. It is a source adapter, not a
live SDK, and uploads only files whose projected content has changed.

## The Evidence Model

Provon organizes tracing data into these levels:

```text
project
  conversation
    trace
      span
        event
```

- A **trace** is one bounded unit of work, such as one agent run or one conversation turn.
- A **span** is one ordered operation inside that work, such as model inference or tool execution.
- A **conversation** groups related traces across turns or requests.
- A **user** is an optional analysis dimension, not part of the parent-child trace tree.
- An **event** records a point-in-time fact inside a span.

Use `gen_ai.conversation.id` to keep all turns of one conversation together. Preserve OTel trace
and parent span IDs to keep causal order inside each trace.

See [Trace model and instrumentation](./tracing-model.md) for boundaries and a diagnosis-ready
trace structure. Use the [Tracing attribute reference](./tracing-attributes.md) for the exact field
contract and [Diagnosis-ready tracing](./tracing-best-practices.md) for the practical quality
checklist.

## From Trace To Finding

Trace ingestion and diagnosis are separate stages:

1. Provon accepts Gateway evidence, OTLP telemetry, or projected transcripts.
2. The trace read model makes spans searchable and groups them into conversations.
3. Enabled diagnostic Rules reconstruct an ordered conversation trajectory.
4. Deterministic signals and optional model adjudication assess supported problems.
5. Candidates above the Rule thresholds become Findings.

This separation is intentional. A successful OTLP response proves only that ingest accepted the
payload. A visible trace proves that the read path is ready. A Finding additionally requires usable
conversation evidence and a supported diagnosis.

## Verify Your First Trace

Open **Traces**, select the new trace, and verify:

- the root span represents the complete unit of work;
- span names are stable operation names rather than request-specific values;
- model and tool spans have the expected parent-child order;
- `gen_ai.conversation.id` is present when work spans multiple traces;
- input, output, tool result, status, token, cost, and error fields appear when emitted;
- sensitive values have been removed before export.

Then investigate the evidence in the [Workbench](./trace-explorer.md), with the
[CLI](./tracing-cli.md), or through the [Trace read API](./trace-api.md).

## Production Next Steps

Before increasing traffic:

- define the trace and conversation boundary;
- set a stable `service.name`;
- decide which content must be retained for diagnosis;
- choose a conversation-consistent sampling strategy;
- batch exports and flush short-lived processes;
- test one known failure from ingest through Finding publication.

See [Production tracing](./tracing-production.md) for sampling, privacy, batching, and retention.

## Documentation Path

Use the tracing documentation in this order:

1. **Start**: complete the [Tracing quickstart](./tracing-quickstart.md).
2. **Capture**: choose [Gateway](./gateway.md), [OTLP](./opentelemetry.md), or
   [transcript sync](./agent-transcripts.md).
3. **Instrument**: define boundaries with the [trace model](./tracing-model.md), select fields from
   the [attribute reference](./tracing-attributes.md), and apply the
   [diagnosis-ready checklist](./tracing-best-practices.md).
4. **Scale the topology**: preserve participant ownership with
   [multi-agent and distributed tracing](./tracing-multi-agent.md).
5. **Investigate**: use the [Workbench](./trace-explorer.md), [CLI](./tracing-cli.md), or
   [Trace read API](./trace-api.md).
6. **Diagnose**: understand [Findings](./findings.md) or run
   [trace diagnosis with CLI](./trace-diagnosis.md).
7. **Operate**: set sampling, privacy, and retention in
   [Production tracing](./tracing-production.md), using the [OTLP API](./otlp-api.md) as the
   transport reference.