Practical guide
What to look for in a verified dataset’s provenance
Questions to ask before trusting the reference answers in an evaluation package.
Identify the authority and the exact record
Start with the source that can establish the fact. A dataset should make the relevant record identifiable, not simply point to an organization’s homepage. Depending on the domain, that might mean a filing accession, a publication identifier, or a versioned source relationship.
A source URL is useful, but it is not the whole audit trail. Review the identity, period, units, and extraction context that connect the record to the reference answer.
Ask when the evidence was collected
Facts and source schemas can change. A retrieval timestamp and preserved evidence help explain what was checked at the time. They do not establish that the source was checked again today.
Treat a historical source snapshot as historical. For time-sensitive uses, agree on refresh requirements and record source failures instead of replacing missing information with guesses.
Inspect the grading rule
A reference answer must be paired with a rule appropriate to the task. Exact dates, numerical tolerance, units, and acceptable answer forms can change how an output is graded.
Inspect borderline cases and raw outputs. If human agreement was not measured, the package should say so. A claim of verification does not justify inventing an inter-annotator agreement score.
Check the package you actually receive
Compare the delivered row count and schema with the product description. Review the license, manifest, duplicate checks, and limitations. Hashes can establish that files match a recorded artifact; they cannot establish that its answers are factually correct.
Groundtruth’s packaged drug proof evaluation includes a manifest, original provenance, historical grades, and explicit usage terms. Those make the package inspectable while preserving the limits of the underlying source and evaluation.