The same 125-million-parameter base, fine-tuned to answer questions with no passage at all — from memory, or not at all. This page is as much a demonstration of where small models break as of what they can do.
Roughly half of this model's confident answers contain invented specifics — figures, dates, names, holdings that are plausible, well-formed, and false. That is measured, not estimated: 48.6% on 327 held-out questions.
This is not a bug in the fine-tune. 125M parameters cannot hold the facts, so the model produces the shape of a confident answer with whatever content it can assemble. It is here to be studied, not consulted.
It knows when a question is unanswerable. Ask “What did the Court hold?” with no case named and it declines, rather than inventing a holding. It does this on 80% of such questions, while declining only 4.6% of answerable ones — so it learned the distinction, not a habit of ducking. The untrained base model never declines anything.
It does not know when it is wrong. The refusal training covers questions that are visibly underspecified. Nothing fires on “what percentage did company X purchase directly” — that looks perfectly well-formed and simply needs a fact the model does not have. So it answers, confidently, with a number it made up.
| answerable (n=327) | correct | partial | wrong | hallucinated |
|---|---|---|---|---|
| base, no fine-tune | 0.3% | 0.6% | 99.1% | 1.5% |
| this model | 4.0% | 16.5% | 79.5% | 48.6% |
| unanswerable (n=30) | declines to answer | |||
|---|---|---|---|---|
| base, no fine-tune | 0.0% | |||
| this model | 80.0% | |||
The base model's low hallucination rate is not a virtue — it barely answers at all. The fine-tune traded incoherence for confident invention. Answers graded by gpt-5.4-mini against reference answers.
7,141 pairs: 4,946 mined from an existing grounded Q&A set by keeping only the questions that still stand alone once the passage is removed, plus 2,195 generated fresh and put through a self-containment gate and a faithfulness gate, plus 714 refusals. Trained 4 epochs on one H100; epoch 2 kept, because the later ones overfit. Total data + training cost: about $12.