Diagnosis-Ready Tracing

Use this guide after sending your first trace. For boundaries, see Trace model and instrumentation. For the field-level contract, use the Tracing attribute reference.

Start With One Representative Run#

Instrument one real workflow before broadening coverage:

1. Choose a run with a clear user goal, at least one model call, and one tool or retrieval step.
2. Capture the complete run, including a failure or negative result when possible.
3. Open the trace in the Workbench and compare it with the application's execution.
4. Fix missing boundaries, ownership, inputs, results, and errors.
5. Confirm that the conversation view tells the same story across turns.
6. Run the relevant diagnostic Rule and inspect the evidence it can support.

Volume does not compensate for weak structure. One complete trace is a better integration test than thousands of model-call records without tool results or a terminal answer.

Define Stable Boundaries#

Use one trace for one bounded unit of work:

• one agent run;
• one conversation turn;
• one request from user input to terminal answer;
• one workflow execution with a defined end.

Use a new trace for each later turn or independent run. Keep gen_ai.conversation.id stable when those traces belong to the same user goal or interaction.

Do not keep an OTel trace open for an unbounded conversation. Long-lived traces are difficult to flush, sample, inspect, and retry, and they obscure which turn introduced a failure.

Make The Root Span Reviewable#

The root span should answer these questions at a glance:

1. What goal or constraint entered the system?
2. What terminal answer or action left the system?
3. Did the bounded run complete successfully?

Use gen_ai.input.messages and gen_ai.output.messages for readable user and assistant content. Use child spans for model, retrieval, tool, delegation, and verification work.

Do not use an HTTP server span as the only root-level explanation when the important unit of work is an agent run. Infrastructure spans can remain in the trace, but the agent operation must be identifiable.

Treat Names As A Query Contract#

Use stable, verb-first names:

invoke-support-agent
classify-intent
retrieve-policy
execute-refund-check
generate-answer
verify-answer

Keep request IDs, users, models, retry numbers, and timestamps out of span names. Put dynamic values in attributes instead.

Stable names keep searches, dashboards, and diagnostic targeting meaningful across releases.

Capture Execution Facts, Not Conclusions#

Record objective evidence on the span where it happened:

• the model and provider used;
• the messages or input that drove the operation;
• tool arguments and the result actually returned;
• retrieved documents or bounded evidence references;
• token, cost, and latency data when available;
• span status and error.type;
• retry and recovery attempts as separate spans;
• the final answer or explicit non-completion.

Do not write derived labels such as hallucinated=true or root_cause=bad_prompt into execution spans as if they were observed facts. Provon Rules derive diagnostic conclusions from the evidence and publish them as Findings.

Preserve Causality#

Parent-child structure should match execution ownership:

invoke-agent
|-- retrieve-context
|-- generate-plan
|-- execute-tool attempt=1 status=ERROR
|-- execute-tool attempt=2 status=OK
`-- verify-answer

• Propagate W3C trace context across service and queue boundaries.
• Keep each failed attempt; do not replace it with the successful retry.
• Use the same gen_ai.tool.call.id to correlate a requested tool call with its result.
• Set the failed span status to ERROR and include a stable error.type.
• Keep verification after the action it verifies.

Timestamps help order events, but they do not replace parent span IDs, tool call IDs, and explicit participant identity.

Attribute Every Participant#

For multi-agent systems, set gen_ai.agent.id on every agent-owned span. Add gen_ai.agent.name and gen_ai.agent.version when available.

Do not flatten several agents into one anonymous orchestration span. A diagnosis must be able to distinguish the participant that delegated work, the participant that acted, and the participant that produced the terminal answer.

See Multi-agent and distributed tracing for topology and handoff patterns.

Keep Useful Evidence, Remove Noise#

Keep a span when it changes how a reviewer understands the run:

• agent and workflow boundaries;
• model inference;
• retrieval;
• tool execution;
• external side effects;
• validation and verification;
• retry, fallback, and recovery;
• errors and policy decisions.

Consider filtering repetitive framework, database, and HTTP internals when they do not explain agent behavior. Preserve them when they establish latency, ownership, or a failure boundary.

Do not duplicate the same prompt or response across every child span. Put evidence on the operation that owns it and use the root span for the bounded run's user-facing input and output.

Bound Structured Content#

Prompts, tool results, retrieved documents, and raw payloads can be both large and sensitive:

• redact credentials and personal data before export;
• truncate or summarize unbounded tool output;
• retain stable references when full documents live elsewhere;
• avoid inline binary content;
• choose payload retention before production traffic;
• test whether the retained projection still supports the enabled Rules.

Redaction after ingestion cannot prevent sensitive data from entering the telemetry pipeline.

Quality Levels#

Aim for diagnosable traces on workflows that feed Findings. Infrastructure-only telemetry can remain visible or searchable without carrying private model content.

Common Failure Patterns#

Review Checklist#

• The trace covers one bounded run or turn.
• The root input states the real user goal.
• The root output records the terminal answer or explicit failure.
• Span names are stable and low-cardinality.
• Model, retrieval, tool, and verification work appear in causal order.
• Every agent-owned span has an explicit participant ID.
• Tool calls, results, failures, and retries remain distinct.
• Errors use span status ERROR and error.type.
• Related traces share one gen_ai.conversation.id.
• Sensitive and unbounded content is removed before export.
• The Workbench trace and conversation views match the application's execution.
• A known failure can support a Finding without relying on unstated assumptions.

Related Docs#

• Tracing overview
• Tracing quickstart
• Trace model and instrumentation
• Tracing attribute reference
• Multi-agent and distributed tracing
• OpenTelemetry setup
• Production tracing
• Findings