# Dataset Best Practices

A well-curated Dataset improves model behavior more reliably than a large but noisy one. These
practices help you collect, review, and maintain Examples that stay useful across training and
evaluation runs.

## Prefer Quality Over Quantity

A smaller coherent Dataset almost always beats a larger contradictory one. Before keeping an
Example, ask:

- Is the input complete enough that the expected output makes sense without extra context?
- Is the expected output actually correct, or is it just the last message the assistant produced?
- Would I be comfortable if this Example were used verbatim in training?

If the answer to any question is no, edit or remove the Example.

## Review Checklist

For each Example before it is used for training or evaluation:

- [ ] **Input preserves the task.** The user message, system prompt, and tool context needed for the
      task are present and not truncated.
- [ ] **Expected output is correct.** It solves the task, follows project conventions, and is free of
      hallucinations.
- [ ] **Rejected output is genuinely worse.** For preference data, the rejected response is a
      plausible but inferior alternative, not an arbitrary wrong answer.
- [ ] **Provenance is intact.** A `conversation` source has trace IDs and a conversation ID; a manual
      or imported source has an external reference.
- [ ] **Privacy boundary is respected.** PII, credentials, tokens, and internal identifiers are not
      present unless they are the explicit subject of the task.
- [ ] **No duplication.** The same evidence and policy revision produces an idempotent Example, but
      manual or imported entries can still duplicate semantically identical cases.
- [ ] **No contradictions.** Similar inputs do not lead to opposite expected outputs.

## Choose The Right Objective

Match the collection objective to the behavior you want to teach:

| If you want to...                 | Use this objective             | Prerequisite                               |
| --------------------------------- | ------------------------------ | ------------------------------------------ |
| Keep behavior that already works  | `preserve_successful_behavior` | Completed, high-quality conversations      |
| Fix a diagnosed failure pattern   | `correct_failed_conversations` | Findings with corrected expected responses |
| Learn from explicit user feedback | `learn_from_user_corrections`  | Rejected answers and verified corrections  |
| Ground responses in project facts | `teach_project_knowledge`      | Reviewed Knowledge Goals                   |

Avoid mixing objectives inside one Dataset. Each Dataset has a single collection policy, and mixed
signals make review and export harder.

## Tag Conventions

Tags are optional but make large Datasets manageable. Consider a small, consistent vocabulary:

| Prefix       | Example                                  | Meaning                             |
| ------------ | ---------------------------------------- | ----------------------------------- |
| `objective:` | `objective:preserve_successful_behavior` | Why the Example was collected       |
| `source:`    | `source:manual`, `source:conversation`   | How the Example entered the Dataset |
| `review:`    | `reviewed`, `review:pending`             | Review state                        |
| `domain:`    | `domain:support`, `domain:api`           | Functional area                     |
| `split:`     | `split:train`, `split:eval`              | Intended use for an evaluation run  |

Keep tags short, lowercase, and hyphen-separated. Tags must be unique within an Example and cannot be
empty.

## Manual Examples vs. Automatic Collection

Automatic collection scales, but it can only capture what already happened in traces. Add manual
Examples when:

- The desired behavior has not yet appeared in production.
- You want to teach a narrow edge case without waiting for it to occur naturally.
- You need a canonical reference that downstream reviewers can compare against.

When adding a manual Example, always set `source.externalRef` or `source.traceIds` so reviewers can
understand where it came from.

## Balance Coverage

A Dataset that only contains the most recent or most common conversations will overfit to recent
behavior. Aim for:

- **Temporal spread.** Include Examples from different time windows.
- **Difficulty spread.** Include both routine and edge cases.
- **Outcome spread.** For `preserve_successful_behavior`, include different successful paths, not just
  the shortest one.

Use the Workbench filters or API list endpoints to inspect the distribution of source kinds, tags,
and creation dates.

## Training And Evaluation Splits

Provon does not split a Dataset into train/eval automatically. When you prepare a fine-tuning or
evaluation run:

- Reserve a held-out evaluation set before sampling for training.
- Use tags such as `split:eval` to mark Examples that should not be consumed by training runs.
- Keep the evaluation set representative of the production distribution you care about.

Fine-tuning materializes a run manifest from the selected Examples, so the split is frozen at run
creation time.

## Privacy And Compliance

Treat a Dataset as a training artifact. Any data inside it can end up in a model checkpoint or
exported JSONL file.

- Remove or mask personal identifiers, API keys, and internal hostnames.
- Review `conversation` sources against your data-retention policy.
- Disable collection when you no longer want new production data entering the Dataset.

## Keep Datasets Focused

One Dataset should teach one thing. Creating separate Datasets for separate behaviors makes it
easier to:

- review Examples with consistent criteria;
- mix and match Datasets in fine-tuning jobs;
- retire outdated behavior without affecting unrelated examples.