Every dataset is a benchmark you can run in an afternoon

Each row ships as a complete, machine-runnable test case: a fair open question, the verified answer re-derived from the primary source, the source URL, grading context, and a frontier model's actual graded attempt as a baseline. Buy a dataset, download the .eval.jsonl variant from the same link, and you have a hallucination benchmark, not just an answer key.

The loop is four steps

1) Loop over the rows and take each one's question. 2) Ask your model cold, no tools, no retrieval, so the answer reveals what it actually knows instead of what it can look up. 3) Grade the answer against expected_answer: exact match for numbers and dates, or one cheap judge-model call for prose. 4) Count the verdicts. Correct, hallucinated, or hedged. The hallucinated percentage is your model's hallucination rate on that domain.

What teams do with the score: catch regressions before shipping a fine-tune, compare model vendors on the domains they actually care about, measure how much their RAG pipeline really helps (run the same questions with retrieval on, and the gap between the two scores is what retrieval buys), and check calibration, because a model that hedges when unsure is safer than one that confidently invents.

Works with any model, any stack

The eval file is plain JSONL, and the runner only needs one thing from your side: a way to send a prompt and get text back. Three ways to plug in, and one of them covers literally anything:

And because the format is plain JSONL with named fields, it also drops into promptfoo, lm-eval-harness, or any in-house harness with a few-line adapter — no lock-in to our runner at all.

A complete runner, small enough to read

This is the core of it. The full version (provider flags, the universal --cmd mode, failure listing, judge-model override) is free: download eval-runner.mjs.

// hallucination check in ~25 lines (Node 18+, no dependencies)
import { readFileSync } from 'node:fs'

const cases = readFileSync('math-verification.eval.jsonl', 'utf8')
  .trim().split('\n').map(l => JSON.parse(l))

let correct = 0, hallucinated = 0, hedged = 0
for (const c of cases) {
  const answer = await askYourModel(c.question)      // your API call, cold: no tools, no retrieval
  const grade = await askYourModel(                  // or a cheap judge model
    'Grade this answer against the verified ground truth. ' +
    'Reply one word: correct, hallucinated, or hedged.\n' +
    'QUESTION: ' + c.question + '\n' +
    'GROUND TRUTH: ' + c.expected_answer + '\n' +
    'CONTEXT: ' + c.grading_context + '\n' +
    'ANSWER: ' + answer)
  if (/halluc/i.test(grade)) hallucinated++
  else if (/hedge/i.test(grade)) hedged++
  else correct++
}
console.log({ correct, hallucinated, hedged,
  hallucinationRate: (100 * hallucinated / cases.length).toFixed(1) + '%' })

The Groundtruth bench: how a frontier model actually scores, cold

We already ran this exact loop for you. Every case below was asked to Claude Sonnet 5 with no tools and no retrieval, and graded against the verified answer. 918 cases across 17 datasets (rates below are from the 874 already-graded cases; the 44 rows added 2026-07-16 ship with fair questions but not-yet-tested baselines): 70.8% correct, 10.1% hallucinated, 18.6% hedged. Every graded response ships inside the datasets, so you can audit our grading row by row.

Read that top line again: on real citation-graph questions, a frontier model asked cold hallucinates roughly one answer in three. That's the gap this data exists to measure.

What one eval case looks like

{
  "id": "MV02",
  "dataset": "math-verification",
  "domain": "math",
  "question": "What is 2^31 - 1, is it prime, and who proved its primality?",
  "expected_answer": "2147483647; prime; proven by Leonhard Euler in 1772",
  "grading_context": "Verified two ways: deterministic BigInt Miller-Rabin and sympy.isprime()...",
  "source_urls": ["https://www.wikidata.org/wiki/Q773522"],
  "reference_model": { "model": "claude-sonnet-5", "verdict": "correct", "response": "..." }
}

Ready to point it at your model? Pick a dataset, or ask us about a custom scope.