Findings
A Finding is Provon's core diagnostic artifact: a supported, actionable problem distilled from agent conversation evidence. It is more specific than a trace error and more durable than an alert.
Use this page for the conceptual model, then see:
- Findings quickstart — first trace-to-Finding flow.
- Diagnostic rules — built-in Rules, 14 finding types, thresholds, and tuning.
- Repair handoff — GitHub, GitLab, Linear, and Jira work items.
- Findings API — HTTP contract for automation.
- Trace diagnosis with CLI — local, non-persisted diagnosis.
- Findings troubleshooting — no findings, false positives, and work-item errors.
Evidence prerequisites#
Findings are conclusions over recorded conversation evidence; they are not generated from a request, a plan, or a closed ticket.
| Prerequisite | Why it matters |
|---|---|
| Traces retained for the project | Diagnosis reads recorded traces and conversation trajectories; telemetry that never arrived cannot be diagnosed |
| Stable conversation identity | Rules operate per conversation; missing or unstable gen_ai.conversation.id fragments evidence |
| At least one enabled Rule | Disabled Rules emit no Findings; all disabled turns automatic scheduling off |
| Conversation settled past the idle window | Automatic runs fire after the inactivity delay (default 10 minutes); a live session is not diagnosed yet |
| Sufficient trajectory evidence | Goal, ordered messages, tool inputs/results, errors, and a terminal outcome; thin traces stay visible but do not qualify |
Review and repair semantics: a model-review proposal is a decision aid, not proof. A healthy conversation may legitimately produce no Finding. A closed issue or an empty inbox does not verify a repair; resolution follows the handoff evidence rules in repair handoff.
Rule, Lead, Finding, Occurrence#
Provon keeps diagnosis in four layers:
| Layer | Responsibility | Persistent result |
|---|---|---|
| Rule | Defines which signals to detect, the publication thresholds, and review policy | Project configuration |
| Signal | One detector output on one conversation, with score and confidence | Consumed during diagnosis; not a stored queue |
| Lead | Credible candidate that did not cross publication thresholds; status: lead, non-scored review tier |
Review Queue output returned to the caller or reviewer |
| Finding | Cluster of supported observations of the same failure mechanism; status: confirmed |
Review and resolution aggregate |
| Occurrence | One row per conversation in which the cluster was observed | Distinct-conversation impact, detection timestamps, latest trace |
A Rule runs over one conversation activity snapshot as an analysis job. The job emits a conversation_analysis_run
trace span; its history (completed, skipped, failed) is projected from that telemetry rather than stored as a
separate record, so there is no standalone Runs page.
Each Finding keeps a bounded occurrence set: the 50 most recent conversations, each with a sorted de-duplicated
detectionTimestamps array, a latestTraceId, and a trimmedDetectionCount for detections dropped by retention
(per conversation capped at 50 timestamps). The impact count is the number of distinct conversations;
firstSeenAt / lastSeenAt are the min/max across detections.
Execution path:
A diagnosis run can complete without publishing a Finding. That is the expected result for healthy conversations or candidates below thresholds.
Workbench: queue and Rules popover#
The Findings page ranks confirmed clusters by diagnostic score. Queue cards switch between All, Active (not resolved and not dismissed), Resolved, and Ignored (dismissed). Leads appear as a separate non-scored review tier rather than as scored Findings.
The Diagnostic rules popover lists the five built-in Rules and exposes these controls:
| Control | Meaning |
|---|---|
| Enable toggle | Run this Rule on qualifying conversations |
| Keywords | Only evaluate conversations containing at least one configured keyword |
Publication confidence (minDiagnosticConfidence) |
Minimum confidence to publish a candidate; default 0.25 |
| Model review toggle + model select | Use model judgment on Rule signals; override the deployment default |
| Signal list | Per-signal enable toggle and severity override per category |
| Lexicon | Locale, built-in lexicon overrides, and extra keyword tables |
Detector settings (detectorSettings JSON) |
Reserved internal field; the control renders but must stay empty — no public detector tuning groups in this build; populated values are rejected |
| Historical scan | Backfill dialog with range presets and conversation budget |
minSignalScore (default 0.25) and reviewerDiscoveryPolicy (candidates_only production default;
whole_trajectory scans the full trajectory and always makes a model call) are configured through the
Findings API, not the popover. Deterministic screening without a model call runs locally
through provon diagnose --mode rule-signals.
The five built-in Rules:
| Rule | Detects |
|---|---|
| Runtime reliability | Runtime interruptions, provider safety/content blocks, degenerate terminal output |
| Tool correctness | Misread tool results, invalid tool preconditions, repetition without progress |
| Task fulfillment | Misunderstood goals, omitted plan steps, insufficient retrieval, capability mismatches, underspecified input |
| Answer grounding | Final-answer claims without supporting evidence |
| Safety | Violated constraints and policies, including prompt injection and secret disclosure |
Automatic and on-demand diagnosis#
Automatic scheduling runs while at least one Rule is enabled; disabling every Rule turns it off.
- New trace activity maps to a stable conversation identifier.
- Enabled Rules are queued for that activity snapshot.
- Each queued diagnosis becomes available after 10 minutes of conversation inactivity (operator-configurable delay; the default is 10 minutes).
- Newer activity supersedes an older automatic snapshot.
- Duplicate diagnoses for the same Rule, conversation, and activity snapshot are not created.
For existing data, open the Rules popover and run an enabled Rule through the historical scan dialog: range presets last 24 hours, 7 days, 30 days, 90 days, or all time; default 30 days; a budget caps conversations per run (default 10,000, maximum 100,000). The API also supports explicit time-window backfills and on-demand discovery.
Reading a Finding#
Expand a row to inspect:
| Field | Meaning |
|---|---|
| Severity | critical, high, medium, low, or info |
| Diagnostic score | Strength and impact used to rank confirmed problems |
| Diagnostic confidence | How strongly available evidence supports the diagnosis |
| Cause assessment | Most likely supported cause |
| Remediation hint | Focused repair direction; not an automatic code change |
| Latest occurrence evidence | Trace, span, or aggregate references with locations and quotes |
| Evidence window | Activity used for the latest occurrence |
| Occurrence count | Distinct conversations observing the same mechanism; expand for per-conversation timestamps and latest trace |
The detail view adds the diagnostic anchor, causal relations (root / symptom / independent), and review
history. Score and confidence are separate: a high-impact problem can have low confidence, and a well-supported
issue can be low severity.
Review and repair lifecycle#
Review state and repair state are deliberately independent:
| Action | Effect |
|---|---|
| Confirm | Accept the Finding as actionable (reviewStatus: confirmed) |
| Dismiss | Move to Ignored with a reason (false_positive, duplicate, accepted_risk, not_actionable, or other); evidence retained; resolution state cleared |
| Create work item | Provider issue created and Finding confirmed in one action |
| Resolve manually | Close without a tracker; resolving also confirms the Finding |
| Retry / Replace / Abandon | Recover failed or disconnected handoffs without deleting history |
| Resolved | Repair verified automatically or explicitly by a user |
| Regressed | A new occurrence appeared after resolvedAt |
Constraints:
- A Finding with an active work item (
todo,doing, orunknown) cannot be dismissed or manually resolved; the API returns409 ACTIVE_WORK_ITEM. - A failed creation is
FindingHandoff.status: failed; it is not a partial work item. - Completing a handoff automatically confirms the Finding.
- Replaying the same activity revision is idempotent and does not increase the occurrence count.
- Historical backfills cannot regress a Finding resolved after that evidence.
Provon never checks out a repository or applies a patch. A user, Claude Code, Codex, or another controlled coding-agent workflow consumes the external work item and performs the repair.
Repair handoff#
GitHub, GitLab, Linear, and Jira are supported. Configure an enabled project connector with a default
destination before creating a work item. Each issue contains the goal, what is wrong, cause assessment,
evidence, verification steps, and hidden Provon correlation metadata; GitHub issues carry the provon-repair
label. See repair handoff for destinations, sync rules, and error codes.
Data requirements#
Diagnosis works best when telemetry preserves:
- a stable
gen_ai.conversation.id; - the user's goal and constraints;
- ordered assistant, model, and tool events;
- tool inputs, results, failures, and retries;
- a terminal answer or explicit non-completion;
- trace and span identifiers for evidence links.
If conversation identity or span evidence is missing, Provon may still show telemetry while skipping diagnosis because no usable trajectory can be reconstructed. See Diagnosis-ready tracing and the trace model.
CLI diagnosis#
provon diagnose <source> runs the same built-in Rule definitions over a trace source. Results return to the
caller only: they are not persisted, do not enter the Findings queue, and cannot create a repair issue. Persisted
Findings are produced by server-side diagnosis over ingested project evidence. See
Trace diagnosis with CLI.
Limitations#
- A healthy or trivial conversation yielding no Finding is correct behavior; it is not a Rules failure.
- Leads are review prompts, not published Findings; they do not carry score ranking or occurrence impact.
- Dismissal hides a cluster from Active queues but keeps evidence; recurrence can surface a new cluster.
- Tracker completion without verified evidence stays in
awaiting_verificationrather thanresolved.