Practical guide
Evaluation, remediation, and validation need different data
How to keep a useful baseline from turning into a misleading improvement claim.
Give each dataset one job
A proof evaluation diagnoses a capability. Remediation data helps a team address the failures it found. Validation asks whether the changed system performs better on cases that did not guide the change.
Using the same questions for all three stages makes the result hard to interpret. A higher score may reflect familiarity with the examples rather than better performance on the underlying task.
Separate more than row IDs
Distinct identifiers do not prove distinct information. Review duplicated questions, shared source records, paraphrases, and repeated entities or periods where these could reveal an answer.
The right split depends on the intended use. Document the split rule and its limitations instead of describing a file as independently held out merely because it has a validation label.
Protect validation during iteration
Do not use final validation cases to choose prompts, retrieval settings, training checkpoints, or preferred interventions. Once results influence those choices, that set is serving as development data.
If a leak or overlap is discovered, record it. Reclassify the affected evidence and design a new evaluation when appropriate; silently changing labels does not restore independence.
Report the experiment, not just the gain
Keep baseline and post-change outputs, model identifiers, settings, grading rules, and failure-family results. Report regressions and unresolved cases alongside improvements.
Groundtruth can help organize the source-backed data for this workflow. Your team applies the intervention. An observed difference is evidence about the tested conditions, not automatic proof of causality or generalization.