# Production Tracing

Production tracing must preserve enough evidence for diagnosis without treating prompts, tool
results, and raw payloads as harmless logs. Make sampling, content capture, and retention explicit
deployment decisions.

## Sampling Strategy

Provon accepts the sampling decision made by the Gateway, SDK, or Collector. It does not reconstruct
dropped spans.

### Preserve Complete Conversations

Provon diagnoses ordered conversation trajectories. Independently sampling each trace can retain one
turn while dropping the earlier goal, failed tool result, or final answer.

For projects that use diagnostic Rules:

- prefer full capture while validating instrumentation;
- make one deterministic keep or drop decision per conversation;
- apply that decision to every trace in the conversation;
- preserve all spans in each sampled trace;
- keep known failures, recoveries, expensive runs, and slow runs.

A standard trace-ID ratio sampler is useful for traffic control, but separate turns usually have
different trace IDs. Use an application-level conversation decision when conversation completeness
matters.

### Head Sampling

Head sampling decides before the operation runs. It is cheap and predictable but cannot know whether
the run will fail.

For distributed traces, use a parent-based sampler so child services honor the root decision:

```ts
import { ParentBasedSampler, TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-base';
import { NodeSDK } from '@opentelemetry/sdk-node';

const sdk = new NodeSDK({
  sampler: new ParentBasedSampler({
    root: new TraceIdRatioBasedSampler(0.2),
  }),
  // Configure the Provon OTLP/HTTP exporter and batch processor.
});
```

This retains approximately 20 percent of root traces. It does not by itself keep all traces in a
multi-trace conversation.

### Collector Tail Sampling

Tail sampling can inspect status, latency, and attributes before deciding. It is useful for keeping
errors and high-value traces while reducing healthy traffic.

Tail sampling still operates on individual OTel traces. If one conversation spans many traces,
combine it with an application-level conversation decision or accept that conversation diagnosis
will be incomplete.

### Gateway Capture Filters

Gateway trace capture supports:

- `all`;
- `off`;
- `filtered` by failed, blocked, or recovered requests, duration, cost, tokens, status code, or
  provider error category.

`filtered` is appropriate for request-level Gateway investigation. It is not equivalent to
conversation-consistent sampling and can remove the healthy or earlier turns needed by diagnostic
Rules.

## Context Propagation

Propagate W3C `traceparent` across synchronous HTTP and RPC boundaries so downstream work joins the
active trace. Use W3C baggage only for bounded, non-sensitive correlation values.

At queue and durable workflow boundaries, decide whether the consumer is a child operation in the
same bounded execution or a new independently scheduled run:

- keep the same trace only when trace context remains valid and the producer waits on that causal
  work;
- start a new trace for resumed, delayed, independently retried, or separately terminated work;
- preserve `gen_ai.conversation.id` and `gen_ai.agent.id` across the new run;
- do not set a parent span ID that belongs to another trace.

See [Multi-agent and distributed tracing](./tracing-multi-agent.md) for participant and handoff
topologies.

## Batching And Delivery

Use a batch span processor in long-running services. For short-lived commands, jobs, and serverless
invocations, flush or shut down the provider before the process exits.

An OTLP `200` response from Provon means the decoded payload was staged and the ingest job was
queued. Trace summaries become visible asynchronously.

Handle ingest responses as follows:

| Status         | Meaning                                                    | Client action                              |
| -------------- | ---------------------------------------------------------- | ------------------------------------------ |
| `200`          | Staged and queued                                          | Do not resend the same batch               |
| `401` or `403` | Invalid key or missing capability                          | Fix authentication                         |
| `402`          | Organization ingest suspended                              | Resolve the usage limit                    |
| `413`          | Decoded payload too large                                  | Reduce exporter batch size                 |
| `422`          | Unsupported content type, encoding, or protobuf capability | Fix exporter configuration                 |
| `501`          | Signal is not enabled by this runtime                      | Enable it or stop exporting that signal    |
| `503`          | Staging unavailable or queue backpressure                  | Retry with backoff and honor `Retry-After` |

The default decoded OTLP request limit is 10 MiB. Runtime configuration can lower or raise it.
Prefer smaller bounded batches over requests close to the limit.

An ingest `200` is asynchronous acceptance, not record-level validation or query readiness. See the
[OTLP/HTTP API](./otlp-api.md) for the exact response, background validation boundary, and retry
contract.

## Content Capture

Prompts, responses, system instructions, tool arguments, tool results, retrieved documents, and
attachments can contain:

- credentials and access tokens;
- personal or regulated data;
- source code and proprietary documents;
- untrusted model output;
- large binary or multimodal payloads.

Redact before export whenever possible. Collector-side processors are useful for centralized
policy, but source-side redaction prevents sensitive data from entering any telemetry queue.

Do not put project API keys, provider credentials, connector tokens, or authentication headers in
span attributes, events, log bodies, or OTel baggage.

## Payload Retention

Worker deployments expose:

```text
PROVON_TELEMETRY_PAYLOAD_RETENTION=all
PROVON_TELEMETRY_PAYLOAD_RETENTION=projected-only
```

`all` retains projected query fields and raw payload evidence. `projected-only` retains projected
fields used for lists, search, summaries, dashboards, and usage queries without full raw payload
bodies.

Projected-only retention reduces exposure and storage, but it can remove the exact message, tool
result, or provider body needed for span inspection and diagnosis. Test the enabled Rules against
the chosen mode before production rollout.

Telemetry data retention and payload retention answer different questions:

- **Data retention** controls how long telemetry rows remain available.
- **Payload retention** controls whether complete raw evidence is stored.

Configure both deliberately. Deleting old projected rows does not replace source-side minimization,
and dropping raw payloads does not remove all identity or usage fields.

## Cardinality And Payload Bounds

- Keep span names low-cardinality and stable.
- Put request-specific IDs in attributes.
- Bound tool outputs and retrieved documents before export.
- Store large supported media as attachments rather than repeated inline values.
- Avoid copying the same prompt or response onto every child span.
- Use opaque user and conversation IDs.

High-cardinality values are valid evidence attributes, but poor operation names and repeated large
payloads make search, storage, and review less useful.

## Environment Isolation

Use separate Provon projects and API keys for production, staging, and development when access,
retention, or diagnostics policy differs. Also emit a stable environment attribute for filtering.

Never reuse a production project API key in local examples or client-side browser code. Rotate
project keys independently from model-provider credentials.

## Release Checklist

- One agent run or turn maps to one bounded trace.
- Conversation sampling keeps all required turns.
- Parent-child context survives service and queue boundaries.
- Every agent-owned span has an explicit participant ID.
- Exporters batch, retry, and flush correctly.
- Errors set span status and `error.type`.
- Private content is redacted before export.
- Tool and message payload sizes are bounded.
- Payload and data retention are configured.
- One known failure remains diagnosable end to end.
- One healthy run produces no false Finding.

## Related Docs

- [Trace model and instrumentation](./tracing-model.md)
- [Tracing attribute reference](./tracing-attributes.md)
- [Diagnosis-ready tracing](./tracing-best-practices.md)
- [Multi-agent and distributed tracing](./tracing-multi-agent.md)
- [OpenTelemetry setup](./opentelemetry.md)
- [OTLP/HTTP API](./otlp-api.md)
- [Trace read API](./trace-api.md)
- [Gateway](./gateway.md)
- [Telemetry architecture](./telemetry.md)
- [Troubleshooting](./troubleshooting.md)