Trace Model And Instrumentation

Use the Tracing attribute reference for exact fields and Diagnosis-ready tracing for the practical review workflow and common anti-patterns.

Boundaries#

One Trace#

Use one trace for one bounded unit of work:

• one agent run against a task;
• one request from user input to terminal answer;
• one conversation turn;
• one workflow execution with a defined start and end.

Do not keep one trace open for an unbounded multi-turn conversation. Use a new trace for each turn or run and keep gen_ai.conversation.id stable across them.

One Conversation#

A conversation groups related traces in time order. Provon prefers:

gen_ai.conversation.id

The normalizer also recognizes conversation.id, conversation_id, provon.conversation_id, thread.id, and thread_id.

A trace without a conversation ID remains visible. For diagnosis, Provon falls back to the trace ID as the trajectory boundary, so cross-trace conversation evidence is unavailable.

One Span#

A span represents one operation with one owner and one outcome. Common GenAI operation names are:

Use child spans for work that is causally inside the parent operation. Use separate traces with the same conversation ID for independent agent runs or later conversation turns.

Recommended Trace Tree#

invoke-agent
|-- classify-intent       gen_ai.operation.name=chat
|-- retrieve-context      gen_ai.operation.name=retrieval
|-- search-knowledge-base gen_ai.operation.name=execute_tool
|-- generate-answer       gen_ai.operation.name=chat
`-- verify-answer         application validation span

The root span should cover the complete unit of work. Model, retrieval, tool, and verification spans should appear in execution order beneath the orchestrating span.

For multi-agent systems, set gen_ai.agent.id on every agent-owned span. Do not rely on the span name to identify the participant.

Keep independently scheduled, resumed, or retried agent runs in separate traces and group them with gen_ai.conversation.id. Use one distributed trace only when all participants are part of the same bounded execution and trace context is propagated across the boundary.

See Multi-agent and distributed tracing for participant, handoff, tool ownership, and queue-boundary patterns.

Stable Names#

Treat span names as a query contract:

• use action names such as retrieve-context and verify-answer;
• keep request IDs, user IDs, model names, and retry numbers out of names;
• place dynamic values in attributes;
• keep names stable across releases so searches and dashboards continue to match.

Use attributes such as retry.number=2, not a span name such as generate-answer-retry-2.

Identity Attributes#

Provon also normalizes user.email, user.full_name, user.hash, user.name, and user.roles. Only send personal data that the deployment is allowed to retain.

GenAI Evidence Attributes#

Provon preserves unknown OTel attributes and projects recognized fields into GenAI read models:

Follow the current OpenTelemetry GenAI semantic conventions where possible. If a language SDK cannot set structured attribute values directly, JSON-encode them. Provon parses valid JSON-looking strings for messages, tool data, definitions, and retrieved documents.

See the Tracing attribute reference for exact field names, accepted conversation aliases, operation mapping, structured-value handling, and cost compatibility fields.

span.setAttribute(
  'gen_ai.input.messages',
  JSON.stringify([{ role: 'user', content: 'Find the failed deployment.' }]),
);
span.setAttribute(
  'gen_ai.output.messages',
  JSON.stringify([{ role: 'assistant', content: 'The canary health check failed.' }]),
);

Tool Evidence#

Use one execute_tool span per attempt. Record:

• a stable gen_ai.tool.call.id;
• gen_ai.tool.name;
• the arguments actually used;
• the returned result or structured error;
• span status ERROR and error.type for failures.

Keep retries as separate spans. Do not overwrite a failed result with the later successful result; diagnostics need both attempts and their order.

Root Input And Output#

The evidence should let a reviewer answer two questions without reconstructing raw provider payloads:

1. What goal or constraint did the user provide?
2. What terminal answer or action did the agent produce?

Put user and assistant content in gen_ai.input.messages and gen_ai.output.messages. Keep raw transport envelopes in separate attributes only when they are needed and allowed by the retention policy.

Errors And Completion#

For a failed operation:

• set the OTel span status to ERROR;
• set error.type to a stable category;
• record the exception event when the SDK supports it;
• keep the failed tool or provider result;
• represent recovery as later spans rather than changing the original fact.

For an incomplete run, end the root span with an error or explicit non-completion result. A final assistant message that claims success after a failed operation is valuable diagnostic evidence and must not erase the underlying error.

Diagnosis Readiness#

Start by making the trace diagnosable. Sampling, redaction, and projected-only retention can intentionally reduce this level; document that trade-off before production rollout.

Related Docs#

• Tracing overview
• Tracing quickstart
• Tracing attribute reference
• Diagnosis-ready tracing
• Multi-agent and distributed tracing
• OpenTelemetry setup
• OTLP/HTTP API
• Production tracing
• Findings