Skip to content

Evals

Evals in the current Provon implementation are workflow evaluator runs, not the canonical diagnostic Rule system. Diagnostic Rules publish persisted Findings through the diagnostic

View as Markdown Open the plain-text version of this page.

The implemented evaluator operation is:

text
eval.evaluator.run

It judges an upstream trace or span subject with either an inline deterministic rule or an LLM judge.

Inputs#

The evaluator requires a trace or span subject from upstream workflow context. If no traceId or spanId is available, the step fails.

Common parameters:

Parameter Required Purpose
judgeName Yes Stable identifier written as gen_ai.evaluation.name
mode Yes llm or rule; defaults to llm in node definitions
passLabel No Label for passing output; default pass
failLabel No Label for failing output; default fail
severity No Default severity for failing output; default medium

LLM Judge Mode#

LLM mode requires prompt. model can override the Gateway model used for the judge when the runtime does not provide a default model.

The prompt should instruct the judge to return strict JSON with score, label, and rationale.

Rule Judge Mode#

Rule mode requires ruleField. ruleOperator defaults to contains.

Supported operators:

text
equals           not_equals       contains
not_contains     matches_pattern  gt
gte              lt               lte

matches_pattern uses wildcard matching: * matches any text and ? matches one character. It is not a regular expression operator.

Outputs#

Evaluator runs produce:

  • workflow context including the evaluator step and score IDs;
  • score artifacts;
  • judgement metadata persisted as gen_ai.evaluation.result;
  • a data record that downstream workflow nodes can route to notifications, issue systems, or data destinations.

Relationship To Findings#

Workflow evals can create or feed Finding-shaped records in automation flows, but diagnostic Rules and server-side Rule Runs remain the canonical source of persisted Findings in the Findings inbox.