Grounded fine-tune · SLM-125M-RAFT
Answer only from the passages, or say so
The base model writes fluent legal prose and invents citations. This fine-tune (Retrieval-Augmented Fine-Tuning, RAFT) is given four passages and a question. It must name the passage, quote it word for word, then answer, or reply “The passages do not contain the answer.” Below: try it live, then see how it was built and how well it works, with every number from the run, including the targets it missed.
125.8M
parameters
the base model, fully fine-tuned
20,340
training examples
teacher-written, checked twice
4
passages per question
the answer is in one, or in none
90%
cite the right passage
of answers on the frozen test set
99.8%
verbatim quotes
with copy-constrained decoding
≈ $14.70
for the whole phase
teacher APIs + 1 × H100
Try it: answer from the passages, or refuse
Live model on a CPU · first call wakes it up (~10–20 s)
SEC filing · answerable. Reference: “Ligand and Allergan”. In the GPU evaluation the model answered — correctly. The live CPU model can differ slightly.
Chapter 1
The problem: grounded or nothing
A pretrained model only continues text. Asked about a filing or a court opinion, it produces something that sounds right, from memory, with no way to check it. For legal and financial text, that is the wrong failure.
The fix is to change the job. Every question arrives with 4 passages. The model must either find the one that answers it and quote it verbatim, or refuse. A wrong refusal is cheap. A confident invented answer is not.
Base model: free text from memory
Photosynthesis is the process by which plants produce more oxygen, carbon dioxide and water from the sun.
Fluent but false, and nothing to check it against.
Answer is in the passages → cite, quote, answer
Q: In the "Kindness" song, what action accompanies the line "I let my Kindness Shine!"?
Source: Passage 2 Quote: "I let my Kindness Shine! (spread hands out from heart, as if presenting)" Answer: spread hands out from heart, as if presenting
The quote must be a verbatim span of the cited passage, so a reader can check it.
Answer is NOT in the passages → refuse
Q: What specific mosquito species is responsible for transmitting the Zika virus in the NE Brazilian microcephaly outbreak?
Source: none Answer: The passages do not contain the answer.
A “near miss”: the passages discuss the same topic but never state this detail.
Takeaway — Grounding turns “what does the model know?” into “can it read these passages?”, which you can check line by line. On the frozen test set, 4.8% of answerable questions were refused.
Chapter 2
One format, one decision token
Every example is one 1,024-token sequence. The prompt holds the passages and the question between the chat tokens <|user|> and <|assistant|>, and ends with Source:. The base model had never seen these tokens, so their embeddings were initialised from the statistics of the trained ones.
The first generated token is the decision: ␣Pass(age k) or ␣none. The loss covers only the target: here 36 of 436 tokens. The model reads the passages but is never trained to reproduce them.
A real training example, token by token
SEC filing · 400 prompt tokens + 36 target tokens. Decision token Trained target
The loss, and what protects the base model
L = LSFT + λ · Lreplay, λ = 1.0
Each 32-row micro-batch carries 26 grounded examples and 6 raw windows of the original pretraining text. Each term is averaged over its own tokens. Without that, the 6 replay windows (≈13K tokens) would outweigh the ≈2.5K target tokens about 5 to 1.
The target, in full
Source: Passage 3 Quote: "There were twenty-four participants in the DTC system holding positions in the Cede certificates." Answer: twenty-four
Citation, verbatim quote, short answer, end of sequence. A refusal target is just Source: none and the fixed sentence.
Takeaway — A single decision token makes refusal a calibrated probability, not a phrasing. Masking the loss to the target means the model learns to read, not to memorise the passages.
Chapter 3
Five kinds of example
A model that only ever sees answerable questions learns to always answer. The training set mixes five kinds of example. About 35% of them should end in a refusal. That share was chosen in the pilot sweep (15 / 25 / 35%), and the final decision threshold τ adjusts the balance afterwards.
- A · Answerable47.2%9,600
- B1 · Oracle removed20.8%4,222
- B2 · Near miss10.8%2,198
- B3 · No / unrelated passages3.5%720
- C · Grounded task17.7%3,600
A · Answerable — One passage states the answer; the other three are distractors (BM25 look-alikes and passages from the same document).
Question: In the "Kindness" song, what action accompanies the line "I let my Kindness Shine!"?
- Passage 1In the Quran, Allah commands believers to take care of orphans, recognizing their vulnerability and the responsibility the community bears towards them. The Importance of Supporting Orphaned Children Fulfilling an Islamic Duty: Supporting orphaned children is not only a noble act of kindness but also a religious obligation in Islam. Allah promises immense rewards for those who care for orphans and treat them with compassion and love. By fulfilling this duty, believers seek the pleasure of Allah and emulate the example set by Prophet Muhammad (peace be upon him).
- Passage 2 · the answer is hereSing the song: “Kindness” together with the moves and/or march around the room or dance: Kindness, Kindness. It feels good to be kind. (give yourself a hug; like hugging a teddy bear) Kindness, Kindness. I let my Kindness Shine! (spread hands out from heart, as if presenting) I’m kind to you. I’m kind to me.
- Passage 3Can be done on Random Acts of Kindness Week in February…or any time. Book Suggestions: “Kids Random Acts of Kindness” by Rosalynn Carter Blues Clues “The Sharing Surprise” by Tish Rabe 1. Have the word KINDNESS written…on a large paper or board. 2. Encourage the children to sound out each letter, and then put the sounds together to form the word. 3. Ask: What does the word Kindness mean to you?
- Passage 4Encourage them to think kindness wishes to themselves, their family, school, community, world. After a moment of quiet, each child is invited to place their gem in the jar. Then, talk about the fact that there are people all over the world who daily express kind wishes to them. (even people they’ve never met) Talk about how that feels to know this. Talk about nurturing our garden of Kindness. Can decorate the jar and have it on display. (as a symbol and reminder of the kindness that’s in every heart) Place a bowl/bag of gems near by.
Target (cite + quote + answer)
Source: Passage 2 Quote: "I let my Kindness Shine! (spread hands out from heart, as if presenting)" Answer: spread hands out from heart, as if presenting
Takeaway — The hard part is the refusals that look answerable: B1 (same question, answer removed) and B2 (near misses). Easy refusals teach almost nothing.
Chapter 4
Teachers, checks & the funnel
A 125M model cannot write its own training data, so a teacher (deepseek-flash, off-peak) wrote questions, quotes and answers from passages of the training corpus. A second model family (gemini-3.8-flash) checked the dev/test labels and audited a sample of the training data.
Nothing a teacher writes is trusted as is. Every candidate is re-checked blind: the checker sees only the passages and the question, and must agree on the passage, the quote and the answer. Of 16,700 calls, 20,340 examples made it into training.
From teacher calls to training examples
Real counts from the 7.8 run. Counts can rise between stages because B1 twins and B3 items are built from accepted questions.
Why questions were dropped (G1, model-free checks)
6,681 of 18,084 written questions dropped before any model check.
The near-miss check (V3)
“Could a careful reader give a plausible answer, including by arithmetic or from a partial statement?” It kept 2,433 of 2,835 near misses (85.8%). The rejected ones were usually answered by a distractor passage.
5% audit by a second model family
| Kind | Audited | Gemini disagrees |
|---|---|---|
| A | 485 | 4.3% |
| B1 | 215 | 0.5% |
| B2 | 123 | 7.3% |
| C | 108 | 6.5% |
The stop line was 15% in any kind.
Leakage rules
- Dev/test come only from held-out documents, scanned for 13-gram overlap with the whole pretraining corpus (< 10%).
- 0 training passages come from a dev/test document.
- 659 training examples were removed because they quoted a document reserved for the “memory” probe.
Takeaway — The two checks that mattered most were model-free (copied wording, vague references) and the dedicated near-miss check. The generic answerability check missed about 1 in 7 near misses that a careful reader could still answer.
Chapter 5
Training without forgetting
Fine-tuning can quietly damage what pretraining built. Two guards were set before training: held-out bits per byte on 600 untouched windows of legal and web text may rise by at most +0.02, and zero-shot CaseHOLD (legal holdings) may fall by at most 2 points.
The first pilot broke both. Measuring after every epoch showed why: forgetting peaks early, while the learning rate is high, then recovers as it decays. Raising the replay weight λ barely helped. Lowering the learning rate fixed it.
Held-out bits/byte change after each epoch (2K-example pilot, 8 epochs)
Five settings of learning rate (LR) and replay weight (λ). One epoch = 39 steps.
- LR 2e-4, λ 0.25
- LR 2e-4, λ 0.5
- LR 2e-4, λ 1
- LR 5e-5, λ 0.5
- LR 5e-5, λ 1 (chosen)
- Limit +0.02
For the chosen setting, CaseHOLD was measured every epoch too: -3.4, -2.3, -4.1, -5.0, -4.1, -4.8, -4.9, -4.7 points. It drops in the first epoch whatever the setting, while bits/byte barely moves.
Takeaway — The learning rate, not the replay weight, controls forgetting here. The final 20K model ends at +0.009 bits/byte (limit +0.02). CaseHOLD fell 4.8 points. A breakdown showed that drop is noisier per-item scoring rather than lost use of context, so it is reported, not gated, by the owner's decision.
Chapter 6
Decision threshold & copy decoding
Because the first token is the decision, the model's willingness to answer is one number: s = P(“none”) / (P(“none”) + P(“Pass”)). It refuses when s ≥ τ. The threshold was chosen on the dev set by a rule fixed in advance: the best balance among values of τ that refuse at least 80% of unanswerable questions and at most 20% of answerable ones. That gave τ = 0.7.
The second fix happens at generation time. Left alone, the model paraphrased inside the quote. Copy-constrained decoding allows only tokens that keep the quote a verbatim span of the cited passage, then forces Answer:. It costs nothing at training time.
Move the threshold: 499 dev questions, scored by the final model
Bars count questions per bin of the decision score s (square-root height, so small bins stay visible). Everything right of the line is refused.
answerable (275)unanswerable (124)
87.1%
of unanswerable questions refused (target ≥ 80%)
B1 95% · B2 67% · B3 100%
1.8%
of answerable questions wrongly refused (target ≤ 20%)
Copy-constrained decoding: same model, same test set
Frozen test set, 1000 items. Plain greedy vs. copy-constrained decoding.
Quote is verbatim
Hallucination rate (lower is better)
Grounded accuracy (judged)
Format valid
Takeaway — Answer-or-refuse works well (test AUROC 0.923), and the constraint took verbatim quotes from 39% to 99.8%. It cannot make the model pick the right span: answer accuracy barely changes.
Chapter 7
More data: the learning curve
The pilot (2,000 examples, 8 epochs) cited the right passage but got only about a fifth of answers right on new questions. Checking it on questions it had trained on settled why: there it was right 95% of the time, against 23% on new ones. It had memorised its examples instead of learning to read.
So the final run used ten times more unique examples, and chose its checkpoint on answer quality, not just the answer-or-refuse decision. Each point below is scored by an independent judge model on the held-out dev set.
Learning curve (dev set, judged)
Training examples on a log scale. 2K is the pilot; 5K, 10K and 20K are the final run's curve.
- Grounded accuracy (judged)
- Hallucination rate
- Grounded accuracy target 60%
Why more data: the pilot memorised
The 2,000-example pilot (8 epochs), checked on 40 questions it trained on and on the held-out dev set.
95%
Questions it trained on: right answer from the right passage
23%
New questions (dev): right answer from the right passage
Takeaway — More unique data helps steadily: about 4.1 points of grounded accuracy per doubling. At that rate, 60% would need on the order of a million examples. For a 125M model, data alone will not close the gap.
Chapter 8
Results, limits & cost
The test set is 1,000 questions from documents the model never trained on, frozen before any training, and scored once. A judge model checked each answer against the reference, and checked that the passages support it. The targets were set in advance and were not lowered afterwards.
The model met 4 of 7 targets. It missed hallucination rate, refusal recall, grounded accuracy (judged).
| Metric | Target | SLM-125M-RAFT | Plain decoding | Teacher ceiling* |
|---|---|---|---|---|
| Hallucination rate | ≤ 15% | 32.5%missed | 53.9% | 1.0% |
| Refusal recall | ≥ 80% | 78.0%missed | 78.0% | 96.2% |
| Over-refusal | ≤ 20% | 4.8%met | 4.8% | 0.0% |
| Grounded accuracy (judged) | ≥ 60% | 31.6%missed | 33.6% | 100.0% |
| Citation accuracy | ≥ 80% | 90.3%met | 89.9% | 100.0% |
| Quote fidelity | ≥ 90% | 99.8%met | 39.1% | 100.0% |
| Format validity | ≥ 99% | 100.0%met | 99.8% | 100.0% |
Also: answer-or-refuse AUROC 0.923; paired consistency 27.3% (answers correctly with the passage and refuses without it, 150 pairs). *Teacher ceiling: deepseek-flash on 200 test items. It is inflated by construction, because items were kept only where the teacher's checks agreed.
What the 32.5% hallucination rate is made of
Bold: share of the hallucinations; grey: share of all test items. Most of it is unsupported content after a citation, not answering the unanswerable.
- Answerable, but the answer is not supported47.1%15.3% of all items
- Grounded task output adds content32.6%10.6% of all items
- Answered a near miss13.5%4.4% of all items
- Answered with the passage removed6.8%2.2% of all items
Refusal recall by kind of unanswerable question
Near misses (B2) are the hard case.
Grounded accuracy by source
Answerable test questions; judged correct and citing the right passage.
What it cost: ≈ $14.70 in total
Teacher APIs (DeepSeek writes and checks, Gemini audits and judges) and GPU time, by step.
Limits, plainly
- It cites the right passage 90% of the time, but the answer is right only about a third of the time. Wrong spans and misread numbers are typical.
- Near-miss questions are refused only 56% of the time.
- The context is 1,024 tokens: at most 4 passages of ~160 tokens.
- Zero-shot CaseHOLD fell 4.8 points after fine-tuning (reported, not gated).
- It is not legal or financial advice. Always check the quote against the passage. That is what the quote is for.
Takeaway — SLM-125M-RAFT is good at deciding whether the passages answer a question, at citing the right passage and at quoting it exactly. It is weak at reading the answer out of the passage. Treat it as a citation finder whose quote you check, not as an answer machine.
Chapter 9
Beyond 125M: what scale buys
SLM-125M-RAFT cites the right passage but reads the right answer out of it only a third of the time. Is that the data, or the model? To find out, the same training rows, the same frozen test set and the same judge were used on three other models, chosen so that size and pretraining change one at a time.
- SLM-500M (same corpus): 4× the parameters, pretrained on the same three sources (case law, SEC filings, web).
- Gemma 3 270M: about our size outside its embeddings (100M vs 113M), but pretrained on ~6T tokens instead of 2.0B unique × 4 epochs.
- Gemma 2 2B: bigger in both ways (2.6B parameters, ~2T tokens).
Grounded accuracy
Frozen test set, judged; target ≥ 60%. Higher is better.
Hallucination
Frozen test set, judged; target ≤ 15%. Lower is better.
Refusal recall
Frozen test set, judged; target ≥ 80%. Higher is better.
| Model | Parameters (outside embeddings) | Pretraining | Fine-tuning | Grounded | Near-miss refusal | Δ bits/byte | Δ CaseHOLD |
|---|---|---|---|---|---|---|---|
| SLM-125M (ours) | 126M (113M) | 2.0B unique × 4 epochs (8.2B seen) | full fine-tuning + replay, 20K rows | 31.6% | 56% | +0.009 | -4.8 |
| SLM-500M (same corpus) | 518M (476M) | 2.2B unique × ~5 epochs (~11B seen) | full fine-tuning + replay, 20K rows | 34.6% | 47% | +0.018 | -1.5 |
| Gemma 3 270M | 268M (100M) | ~6T tokens | full fine-tuning (forgetting gate reported, not met), 20K rows | 64.4% | 79% | +0.308 | -9.6 |
| Gemma 2 2B | 2.6B (2.0B) | ~2T tokens | LoRA r 32, 5K rows | 91.2% | 76% | +0.020 | -0.1 |
Δ bits/byte: change in the model's own held-out bits per byte on our text after fine-tuning (limit +0.02, red when over). Δ CaseHOLD: change in zero-shot legal-reasoning accuracy, in points (reported, not gated).
Reading vs. refusing
Before any fine-tuning, Gemma 2 2B (instruction-tuned, two examples in the prompt) already reads well: grounded accuracy 83.8%. But it refuses only 25% of unanswerable questions and 3% of near misses. After LoRA on 5,000 rows: refusal recall 90%, hallucination 8.6%. Fine-tuning taught it when not to answer. The reading came from pretraining.
Fair test? Checked.
- The 500M splits validation by window, so it probably saw our test documents. Measured against text neither model saw, it showed no extra familiarity with them, and it answered under 1% of test questions with no passages.
- Without passages, Gemma 2 answered 13.8% of test questions correctly. Excluding those, its grounded accuracy barely changes.
- Near misses (the passages almost answer) are the weakest point for every model.
- Copy-constrained decoding keeps small models' quotes verbatim (39% → 100% for ours). For larger models, which already quote accurately, it slightly lowers accuracy.
Takeaway — More parameters on the same 2B-token corpus barely moved the answer quality (+3.0 points). Far larger pretraining at about the same size did (+32.8 points), at a real cost in forgetting. Reading comes mostly from pretraining scale. Fine-tuning mostly teaches the model when to refuse.