Multi-Agent And Distributed Tracing

• the physical execution structure: which process or agent ran which operation;
• the logical conversation structure: which runs contributed to the same user goal.

Provon uses OTel trace and parent span IDs for execution causality, gen_ai.conversation.id for cross-trace grouping, and gen_ai.agent.* for participant attribution. Do not collapse those dimensions into one identifier.

Choose The Topology#

Independent Agent Runs#

Use separate traces when agents are independently scheduled, retried, resumed, or executed at different times:

conversation: repair-42

trace A: planner run
trace B: implementation agent run
trace C: review agent run

Every trace uses the same gen_ai.conversation.id=repair-42, but keeps its own trace ID and bounded root span. This is the default for durable workflows, queued handoffs, later turns, and coding-agent sessions.

Physical separation prevents one participant's retries, status, and terminal answer from being mistaken for another participant's execution.

One Distributed Run#

Use one trace only when all participating services and agents are part of the same bounded causal execution and W3C trace context is propagated across every boundary:

invoke-workflow
|-- invoke-agent planner
|   `-- generate-plan
|-- invoke-agent researcher
|   `-- execute-tool search
`-- invoke-agent writer
    `-- generate-answer

Each agent-owned span still needs its own gen_ai.agent.id. Sharing a trace ID does not provide participant attribution.

Decision Rule#

Do not create parent-child relationships across different trace IDs. An OTel parent span ID is meaningful only inside its trace.

Participant Identity#

Set participant attributes on every agent-owned span:

Example:

span.setAttribute('gen_ai.operation.name', 'invoke_agent');
span.setAttribute('gen_ai.agent.id', 'security-reviewer');
span.setAttribute('gen_ai.agent.name', 'Security Reviewer');
span.setAttribute('gen_ai.agent.version', '2026-08-01');
span.setAttribute('gen_ai.conversation.id', 'repair-42');

Use stable IDs such as planner, security-reviewer, or a configured agent definition ID. Do not use a process ID, request ID, or generated span ID as the agent ID.

If several instances run the same agent behavior, keep one agent ID and put instance identity in a separate operational attribute.

Model Handoffs As Evidence#

Represent a handoff with an invoke_agent span owned by the invoked participant:

invoke-workflow
|-- invoke-agent planner
|   `-- generate-plan
`-- invoke-agent implementer
    |-- execute-tool edit-file
    `-- execute-tool run-tests

Record the delegated goal in gen_ai.input.messages and the participant's returned result in gen_ai.output.messages. Preserve the parent-child relationship when the handoff is synchronous.

For an asynchronous handoff:

1. end the delegating trace with the handoff request as objective evidence;
2. start a new trace when the target participant begins;
3. reuse the conversation ID;
4. put the delegated goal on the new root input;
5. retain an application correlation ID as a normal attribute when needed.

Do not encode an entire handoff protocol into span names. Names should remain stable operations such as delegate-research or invoke-reviewer.

Tool Ownership#

Use one execute_tool span per tool attempt and nest it beneath the agent operation that requested it:

invoke-agent researcher
|-- generate-query
|-- execute-tool search attempt=1 status=ERROR
|-- execute-tool search attempt=2 status=OK
`-- generate-summary

Set:

• gen_ai.tool.name;
• gen_ai.tool.call.id;
• gen_ai.tool.call.arguments;
• gen_ai.tool.call.result;
• span status ERROR and error.type on failures.

Provon attributes the tool event to the tool and uses the parent chain to preserve which agent invoked it. A shared tool call ID does not replace the parent-child relationship.

Conversation Identity#

Use one conversation ID for all traces needed to understand the same user goal. Good boundaries include:

• one chatbot thread;
• one support case;
• one repair task from Finding to review;
• one multi-agent report or workflow instance.

Do not use:

• a global constant shared by all users;
• the trace ID copied into every later trace;
• the agent ID;
• an email address or other unnecessary personal identifier.

Provon prefers gen_ai.conversation.id. It also recognizes conversation.id, conversation_id, provon.conversation_id, thread.id, and thread_id.

Distributed Context#

For synchronous calls, propagate the standard W3C traceparent header so downstream services join the active trace. Propagate only bounded, reviewed values through W3C baggage.

For queues and durable workflows:

• preserve the originating trace and span IDs in job metadata for external correlation;
• start a new trace when the queued run is an independent execution;
• carry the stable conversation ID and participant ID explicitly;
• keep enqueue, dequeue, retry, and dead-letter facts visible when they explain behavior.

Current Provon span projection preserves trace IDs, parent span IDs, events, and attributes, but not OTel span links. Do not rely on span links alone for participant attribution or diagnostic causality.

Failure And Recovery#

Keep failures local to the participant that experienced them:

trace B: implementation agent
|-- execute-tool apply-patch status=OK
|-- execute-tool test status=ERROR
`-- output: implementation incomplete

trace C: review agent
`-- output: rejected because tests failed

Do not mark the whole conversation successful only because a later agent produced an answer. Preserve:

• the failed participant span;
• the error type and result;
• later recovery as a separate span or trace;
• each participant's terminal output;
• the final workflow outcome.

This lets diagnostic Rules distinguish an execution failure, an ignored negative result, a failed handoff, and a false claim of success.

Gateway Context#

For Gateway-captured model calls, pass participant and conversation context as headers:

-H "x-otel-gen-ai-conversation-id: repair-42" \
-H "x-otel-gen-ai-agent-id: security-reviewer" \
-H "x-otel-gen-ai-agent-name: Security Reviewer" \
-H "x-otel-gen-ai-agent-version: 2026-08-01" \
-H "x-otel-gen-ai-workflow-name: repair-workflow"

Gateway capture observes the model boundary. It cannot create application-side tool, handoff, queue, or verification spans that it never sees. Use direct OTLP instrumentation for the orchestration tree when those operations matter to diagnosis.

Review Checklist#

• Every independently scheduled agent run has its own bounded trace.
• Related traces share one stable gen_ai.conversation.id.
• Every agent-owned span has gen_ai.agent.id.
• Agent IDs describe behavior, not process instances.
• Synchronous child work has valid parent-child relationships.
• No parent span ID points into another trace.
• Handoff inputs and returned results are retained.
• Tool spans are nested under the invoking participant.
• Failed and recovered attempts remain separate.
• The final answer can be attributed to one participant.
• Conversation evidence does not merge unrelated users or goals.
• Diagnosis does not depend only on OTel span links.

Related Docs#

• Diagnosis-ready tracing
• Trace model and instrumentation
• Tracing attribute reference
• Gateway evidence
• OpenTelemetry setup
• Production tracing