Skip to content

Dataset Troubleshooting

You created a Dataset, sent traces, but the Examples list stays empty.

View as Markdown Open the plain-text version of this page.

No Examples Were Collected#

You created a Dataset, sent traces, but the Examples list stays empty.

1. Wait For Ingestion And Collection#

Automatic collection is asynchronous. After a conversation is inactive for roughly ten minutes, the runtime evaluates eligible evidence. Check again after the delay plus a few minutes for job processing.

2. Verify Collection Is Enabled#

A Dataset only collects when collectionEnabled is true and it has a collectionPolicy.

bash
provon datasets get <dataset-id>

If collection is disabled, re-enable it:

bash
provon datasets update <dataset-id> --collection-enabled true

3. Check The Objective Requirements#

Objective Required conversation state
preserve_successful_behavior Conversation must be completed.
correct_failed_conversations Conversation must have a diagnosed failure and a corrected response.
learn_from_user_corrections Conversation must contain a rejected answer and a verified correction.
teach_project_knowledge Active Knowledge Goals must exist and the conversation must contain matching items.

Not every trace matches every objective. A completed, uneventful conversation will not produce Examples for correct_failed_conversations.

4. Confirm The Trace Was Ingested#

Look up the conversation or trace in the Workbench under Tracing. If the trace is missing, check:

  • the OTLP endpoint or Gateway configuration;
  • whether the project id matches the Dataset's project;
  • trace attribute requirements for the producer you are using.

5. Verify The Schema#

If the Dataset schema does not match the producer output, the Example will be discarded. Currently only chat schema v1 is supported.

Export Or Fine-Tuning Fails#

input must be chat#

Formats such as openai_chat_sft and preference_chat_dpo require payload.input.type to be chat. Convert the Example or choose a different format.

expectedOutput must be an assistant chat_message#

SFT and DPO formats need an assistant message as the expected output. If your Example stores text or JSON output, either change the Example or use chat_prompt_completion / alpaca_instruction / evaluation_jsonl.

rejectedOutput must be an assistant chat_message#

preference_chat_dpo requires a rejected assistant message in addition to the chosen one. Add a valid rejectedOutput or use a non-preference format.

expectedOutput and rejectedOutput must differ#

For preference formats, the chosen and rejected messages cannot be identical. Verify the Example and update one of the outputs.

expectedOutput is required#

chat_prompt_completion and alpaca_instruction require an expected output. Add one or exclude the Example from the export.

API Errors#

Legacy top-level input/expectedOutput/metadata fields are not accepted#

Example create and update requests must nest input, expectedOutput, and rejectedOutput under payload. Do not send them at the top level of the request body.

source.conversationId is required for conversation sources#

A conversation source must include conversationId and non-empty traceIds.

source.externalRef is required for import sources#

An import source must include an externalRef so the Example remains auditable.

tags must be an array of non-empty strings#

Tags cannot be empty strings and cannot contain duplicates. Remove empty entries before sending.

Fine-Tuning Job Does Not See New Examples#

Fine-tuning jobs materialize an immutable manifest at creation time. Adding or editing Examples after the job starts does not affect it. Cancel the job and create a new one, or wait for it to finish and start a follow-up job with the updated Dataset.

Example Count Looks Wrong#

The exampleCount field is updated in the same transaction as Example create and delete operations. If it seems stale:

  • Refresh the Dataset detail page or re-fetch via the API.
  • Check whether automatic collection produced idempotent updates that did not change the count.
  • Verify that no failed API requests left partial state.

Still Stuck?#

  • Review the Dataset quickstart for a working end-to-end example.
  • Read the export format reference to confirm your Examples match the chosen format.
  • Inspect the conversation in Tracing to confirm it satisfies the objective requirements.