Trace Diagnosis With CLI

trace source -> deterministic signals -> optional Agent adjudication -> Findings

The command does not start or configure a model runtime. The calling Agent supplies semantic judgment through a hash-bound structured response.

The CLI can run on a developer machine, in CI, in a container, or as a cloud job. source describes the evidence input available to that process; it does not describe where diagnosis executes.

Execution Boundary#

Source-based diagnosis:

• does not require a Provon API key;
• does not upload the source;
• does not persist a Finding;
• does not create a GitHub issue;
• does not modify the repository or source files.

It returns evidence-backed Finding objects to stdout. Server-side Rule Runs and the Findings inbox remain a separate remote workflow.

Supported Inputs#

provon diagnose <source>

source can be:

Input formats:

auto
transcript
otlp-json
otlp-protobuf
provon-spans

Automatic detection is the default:

provon diagnose ./trace-evidence
provon diagnose ~/.claude/projects/example/session.jsonl
provon diagnose ./otlp-traces.json
provon diagnose ./traces.pb

Override detection when stdin or an ambiguous extension is involved:

cat ./spans.jsonl | provon diagnose - --input-format provon-spans
provon diagnose ./capture.bin --input-format otlp-protobuf

The same trace/span identity appearing more than once is deduplicated before diagnosis.

Rule Selection#

By default, all five built-in Rules run:

runtime-reliability
tool-correctness
task-fulfillment
answer-grounding
conversation-health

Select one or more Rules by repeating --rule:

provon diagnose ./trace-evidence \
  --rule runtime-reliability \
  --rule tool-correctness

Override publication thresholds for this execution:

provon diagnose ./trace-evidence \
  --min-score 0.5 \
  --min-confidence 0.7

Both values must be between 0 and 1.

Deterministic-Only Mode#

provon diagnose ./trace-evidence --deterministic-only

This mode:

• returns only Findings supported without semantic adjudication;
• emits no adjudication request;
• makes no model call;
• is predictable for CI and high-volume screening;
• intentionally has lower recall for grounding, fulfillment, and causal questions.

Do not treat --deterministic-only as equivalent to complete diagnosis.

Signal Adjudication#

Without --deterministic-only, the first pass can return:

{
  "ok": true,
  "data": {
    "status": "needs_adjudication",
    "findings": [],
    "adjudicationRequests": [
      {
        "schemaVersion": "provon-signal-adjudication-v1",
        "diagnosisInputHash": "...",
        "instructions": "...",
        "evidenceBrief": "...",
        "signals": [],
        "evidence": [],
        "responseContract": {}
      }
    ]
  },
  "warnings": [],
  "nextActions": [
    {
      "label": "Finalize with current Agent adjudication",
      "command": "provon diagnose ./trace-evidence --adjudication-response <response.json>"
    }
  ]
}

The request contains the only candidates the Agent may confirm. The Agent must judge whether each candidate is:

• a genuine fault;
• the earliest uncorrected cause;
• not corrected or compensated for by a later step.

It must not invent a new candidate or use trace IDs, span IDs, conversation IDs, or timestamps as the signalSignature.

Response Shape#

Create one response per diagnosisInputHash. The response can be a single object or a responses envelope:

{
  "responses": [
    {
      "schemaVersion": "provon-signal-adjudication-v1",
      "diagnosisInputHash": "copy-from-the-request",
      "outcome": "failed",
      "confidence": 0.92,
      "signals": [
        {
          "failureModeKind": "goal_not_completed",
          "signalSignature": "copy-a-candidate-signature",
          "severity": "high",
          "confidence": 0.9,
          "title": "Required verification was skipped",
          "summary": "The agent claimed completion without running the required verification.",
          "rootCauseSummary": "The first uncorrected fault is the omitted verification step.",
          "suggestedFix": "Run and report the required verification before claiming completion.",
          "failedActor": "implementation-agent",
          "causeStep": "Claimed completion before verification",
          "causeStepIndex": 8,
          "genuineFault": true,
          "earliestUncorrected": true,
          "notCorrected": true,
          "causalCheckReason": "No later step performed or compensated for the verification."
        }
      ]
    }
  ]
}

Use values from the request's supplied signals. A response with an unknown failure mode or non-matching signal signature cannot confirm that candidate.

Run the same diagnosis again with the response:

provon diagnose ./trace-evidence \
  --adjudication-response ./adjudication-response.json

The command recomputes the diagnosis input and requires the response hash to match. A changed trace, Rule, threshold, or evidence projection invalidates stale adjudication.

Output#

Every successful CLI command returns an { ok, data, warnings, nextActions } envelope. data contains:

Parse and rejected-record messages appear in the envelope's warnings. When adjudication is required, nextActions includes the follow-up command.

Use --include-candidates to inspect promoted, suppressed, pending, and adjudication-rejected candidates:

provon diagnose ./trace-evidence --include-candidates

This is useful for tuning thresholds and understanding why a signal did not become a Finding.

CI Pattern#

provon diagnose ./artifacts/agent-trace.jsonl \
  --input-format provon-spans \
  --deterministic-only \
  --min-score 0.5 \
  --min-confidence 0.7

CI should consume the structured JSON result rather than scrape human text. Decide explicitly which severities or Finding counts fail the job; Provon does not impose a repository-specific gate.

Evidence Requirements#

Diagnosis needs more than valid OTLP:

• a stable conversation boundary;
• the user's goal and constraints;
• ordered model, tool, retrieval, and participant activity;
• tool arguments, results, failures, and retries;
• a terminal answer or explicit non-completion;
• trace and span IDs for evidence references.

When the input contains spans but no diagnosable trajectory, the command fails with diagnose.no_trajectories.

Related Docs#

• Tracing quickstart
• Tracing CLI
• Agent transcript sync
• Diagnosis-ready tracing
• Findings