Skip to content

What Provon Does

Provon turns real agent traces into actionable fixes, reusable Knowledge, reviewed Datasets, and model improvements. Every output remains grounded in the production behavior that m

View as Markdown Open the plain-text version of this page.
flowchart LR
  agent["Agent"] --> capture["Capture traces"]
  capture --> understand["Understand behavior"]
  understand --> findings["Findings"]
  understand --> knowledge["Knowledge"]
  understand --> datasets["Datasets"]
  findings --> repair["Repair handoff"]
  knowledge --> improve["Improve prompts and workflows"]
  datasets --> train["Evaluate or fine-tune"]
  train --> deploy["Deploy through the Gateway"]
  deploy --> agent

Core Loop#

Stage Provon capability Result
Capture Gateway traces, OTLP/HTTP ingest, and transcript projection Project-scoped evidence of what the agent actually did
Understand Trace search, conversation reconstruction, dashboards, and Rules Reviewable behavior, performance, cost, and failure context
Improve Findings, Knowledge extraction, and Dataset collection Evidence-backed inputs for repair, evaluation, and training
Act Issue-system handoff, local model runtime, and fine-tuning A repaired harness or a new model checkpoint
Verify New production traces tied to the same observable behavior Evidence that the change worked or regressed

Provon does not silently modify an application or accept generated conclusions as truth. Users retain control of repair, review, training, and deployment decisions.

Evidence Capture#

Gateway#

The OpenAI-compatible Gateway can:

  • route requests to cloud, custom, and local/private model targets;
  • manage project-scoped provider credentials and model access;
  • apply retries, fallback, load balancing, affinity, circuit breaking, and routing policies;
  • enforce request, token, and cost controls before an upstream call;
  • run data, tool-use, and context guardrails;
  • capture request, response, attempt, token, cost, latency, error, and guardrail evidence.

Use Gateway capture when Provon should own the model-call path.

See the Gateway overview, API contract, routing model, governance controls, and trace evidence.

OpenTelemetry#

Provon accepts standard OTLP/HTTP traces, logs, and metrics in protobuf or JSON. It normalizes GenAI semantics and projects telemetry into trace, conversation, user, cost, latency, and diagnostic views.

Use OTLP when the agent already has instrumentation or must keep calling its provider directly.

Agent Transcripts#

The CLI can project persisted Claude Code, Codex, and Pi transcripts into canonical OpenTelemetry spans and upload them through the same evidence path. Transcript adapters own format detection, parsing, and sanitization; the runtime owns filesystem discovery and upload.

Diagnostics#

Provon reconstructs an ordered conversation trajectory, derives facts, detects deterministic signals, and uses model adjudication only where semantic judgment is required.

The built-in Rules cover:

  • runtime reliability;
  • tool correctness;
  • task fulfillment;
  • answer grounding;
  • conversation health.

Projects can configure keywords, minimum signal score, minimum confidence, adjudication mode, adjudication model, individual signal enablement, and severity overrides.

Automatic diagnosis runs after a conversation is inactive. Backfills and on-demand Runs cover existing telemetry.

Findings#

A Finding contains:

  • a review state and severity;
  • diagnostic score and confidence;
  • a concise problem statement;
  • a supported cause assessment;
  • a remediation hint;
  • trace, span, aggregate, location, and quote evidence;
  • the evidence window and repeat detection count;
  • an optional repair handoff reference and external resolution.

Users can confirm, dismiss, or create a repair Issue from a Finding. Similar repeat observations update the same Finding instance rather than creating an unbounded stream of alerts.

Repair Handoff#

GitHub is the current repair handoff and status-tracking implementation.

Provon creates an Issue containing the repair goal, evidence, cause assessment, remediation direction, delivery requirement, and correlation metadata. A user-controlled coding-agent workflow can then open a pull request. Provon records a merged pull request or closed Issue as the external resolution.

Other issue-system connectors can be configured for repair handoff, but they do not currently replace the GitHub-specific Finding repair-status path.

Supporting Capabilities#

Observability Views#

  • Trace lists, search, waterfalls, span detail, and attachments
  • Conversation and observed-user summaries
  • Token, cost, latency, provider, model, error, and status projections
  • Configurable dashboards over normalized telemetry
  • Project retention policies

Playground#

  • Import a production trace into a replayable session
  • Inspect runnable steps and evidence
  • Compare prompt, model, and parameter variants
  • Record rerun trace references for comparison

Self-Hosted Models And Fine-Tuning#

  • Import models from Hugging Face Hub into the Node runtime
  • Install inference engines (llama.cpp, vLLM, SGLang) and download artifacts
  • Start and stop model services behind self/ Gateway targets
  • Train LoRA and full fine-tuning jobs from chat Datasets
  • Register checkpoints, view lineage, and deploy checkpoints to the Gateway
  • Export checkpoints to Hugging Face Hub in Safetensors or GGUF formats

Self-hosted model execution requires the Node runtime and the optional services/python-inference and services/python-fine-tuning services. It is not available in the Cloudflare runtime.

Knowledge Extraction#

  • Define project goals with free-form extraction instructions
  • Automatically extract structured knowledge items from new agent conversations
  • Backfill existing traces to populate knowledge from historical evidence
  • Review items with confidence scores and trace evidence references
  • Archive or update extracted items as the project learns

Datasets#

  • Collect Examples under one explicit improvement objective
  • Preserve trace and conversation provenance for derived Examples
  • Review canonical inputs, expected outputs, rejected outputs, and rubrics
  • Export one Dataset for evaluation, preference learning, or supervised fine-tuning
  • Freeze immutable manifests so a run remains reproducible after the Dataset changes

Connectors#

  • Project-scoped connectors store encrypted credentials for issue-system handoff and notifications.
  • GitHub is the canonical Finding repair handoff and status-tracking destination.
  • GitLab, Linear, and Jira connectors can create work items from Findings.
  • Slack, Microsoft Teams, and Lark notification connectors can forward Finding review updates.

Connectors support the trace-to-repair handoff. Diagnostic Rules and Findings remain the canonical trace-to-repair path.

Gateway Governance#

  • Provider and model capability discovery
  • Provider key and model binding management
  • Usage reservation and settlement
  • Request, token, and cost policies
  • Guardrails for sensitive data, tool behavior, and context growth
  • Cloud and local/private model targets behind one API contract

Product Boundary#

Provon is not:

  • an agent framework;
  • a replacement for OpenTelemetry;
  • a coding agent that edits repositories;
  • a general-purpose ticketing or enterprise integration platform;
  • a guarantee that every trace problem should become a Finding.

Its responsibility is narrower: preserve production evidence, publish supported problems, and make agent improvement reviewable from the original behavior through the resulting repair or model.