Skip to content

Dataset Quickstart

Create a trace-backed Dataset, collect examples for one objective, and prepare it for evaluation or fine-tuning.

View as Markdown Open the plain-text version of this page.

Prerequisites#

  • A Provon project with conversation traces
  • Evidence that includes user input and assistant output
  • Findings or Knowledge when required by the selected collection objective

1. Create A Dataset#

Open Datasets, select Create Dataset, and provide:

  • a name that describes the intended behavior;
  • an optional description;
  • one collection objective;
  • optional Knowledge Goal filters when teaching project knowledge.

The supported objectives are:

Objective Collects
Preserve successful behavior Verified production conversations worth retaining
Correct failed conversations Diagnosed failures paired with corrected responses
Learn from user corrections Rejected answers paired with verified follow-up responses
Teach project knowledge Examples grounded in selected project Knowledge Goals

Collection is enabled when the Dataset is created. Disable it when you want to freeze the working set for review.

2. Produce Representative Evidence#

Run the agent on tasks that match the Dataset objective. Automatic collection evaluates eligible conversation activity and adds canonical Examples with stable provenance.

Collection is idempotent for the same evidence and policy revision. Reprocessing the same conversation does not create an unbounded stream of duplicates.

3. Review Examples#

Open the Dataset and inspect each Example:

  • confirm that the input preserves the task and required context;
  • verify the expected output;
  • inspect rejected output or rubric data when present;
  • follow trace references back to the source evidence;
  • remove private, unsupported, or contradictory examples.

Manual Examples can be added when the desired input and output are already known. Keep a source reference when the example originated outside Provon.

4. Use The Dataset#

A reviewed Dataset can be:

  • exported in a supported training or evaluation format;
  • selected by a fine-tuning job;
  • consumed by an evaluation workflow without changing the Dataset contract.

Fine-tuning creates an immutable manifest from the selected Examples. Subsequent Dataset edits do not change an existing run.

Next Steps#