Skip to content
SLM-125M

Grounded fine-tune · SLM-125M-RAFT

Answer only from the passages, or say so

The base model writes fluent legal prose and invents citations. This fine-tune (Retrieval-Augmented Fine-Tuning, RAFT) is given four passages and a question. It must name the passage, quote it word for word, then answer, or reply “The passages do not contain the answer.” Below: try it live, then see how it was built and how well it works, with every number from the run, including the targets it missed.

  • 125.8M

    parameters

    the base model, fully fine-tuned

  • 20,340

    training examples

    teacher-written, checked twice

  • 4

    passages per question

    the answer is in one, or in none

  • 90%

    cite the right passage

    of answers on the frozen test set

  • 99.8%

    verbatim quotes

    with copy-constrained decoding

  • ≈ $14.70

    for the whole phase

    teacher APIs + 1 × H100

Try it: answer from the passages, or refuse

Live model on a CPU · first call wakes it up (~10–20 s)

Copy-constrained
4 passages · max 4

SEC filing · answerable. Reference: “Ligand and Allergan”. In the GPU evaluation the model answered — correctly. The live CPU model can differ slightly.

Chapter 1

The problem: grounded or nothing

A pretrained model only continues text. Asked about a filing or a court opinion, it produces something that sounds right, from memory, with no way to check it. For legal and financial text, that is the wrong failure.

The fix is to change the job. Every question arrives with 4 passages. The model must either find the one that answers it and quote it verbatim, or refuse. A wrong refusal is cheap. A confident invented answer is not.

Base model: free text from memory

Photosynthesis is the process by which plants produce more oxygen, carbon dioxide and water from the sun.

Fluent but false, and nothing to check it against.

Answer is in the passages → cite, quote, answer

Q: In the "Kindness" song, what action accompanies the line "I let my Kindness Shine!"?

Source: Passage 2
Quote: "I let my Kindness Shine! (spread hands out from heart, as if presenting)"
Answer: spread hands out from heart, as if presenting

The quote must be a verbatim span of the cited passage, so a reader can check it.

Answer is NOT in the passages → refuse

Q: What specific mosquito species is responsible for transmitting the Zika virus in the NE Brazilian microcephaly outbreak?

Source: none
Answer: The passages do not contain the answer.

A “near miss”: the passages discuss the same topic but never state this detail.

Takeaway — Grounding turns “what does the model know?” into “can it read these passages?”, which you can check line by line. On the frozen test set, 4.8% of answerable questions were refused.

Chapter 2

One format, one decision token

Every example is one 1,024-token sequence. The prompt holds the passages and the question between the chat tokens <|user|> and <|assistant|>, and ends with Source:. The base model had never seen these tokens, so their embeddings were initialised from the statistics of the trained ones.

The first generated token is the decision: ␣Pass(age k) or ␣none. The loss covers only the target: here 36 of 436 tokens. The model reads the passages but is never trained to reproduce them.

A real training example, token by token

SEC filing · 400 prompt tokens + 36 target tokens. Decision token Trained target

Loss mask
<|eos|><|user|>Passage 1: Changes in and Disagreements with Accountants on Accounting and Financial Disclosure There was no change of accountants or disagreement with accountants on any matter of accounting principles or practices or financial disclosure. Item 10. Directors and Executive Report of Independent Auditors To be supplied upon receipt by the Trustee↵Question: According to the SEC filing, how many DTC system participants held positions in the Cede certificates?<|assistant|>Source: Passage 3↵Quote: "There were twenty-four participants in the DTC system holding positions in the Cede certificates."↵Answer: twenty-four<|eos|>

The loss, and what protects the base model

L = LSFT + λ · Lreplay, λ = 1.0

Each 32-row micro-batch carries 26 grounded examples and 6 raw windows of the original pretraining text. Each term is averaged over its own tokens. Without that, the 6 replay windows (≈13K tokens) would outweigh the ≈2.5K target tokens about 5 to 1.

The target, in full

Source: Passage 3
Quote: "There were twenty-four participants in the DTC system holding positions in the Cede certificates."
Answer: twenty-four

Citation, verbatim quote, short answer, end of sequence. A refusal target is just Source: none and the fixed sentence.

Takeaway — A single decision token makes refusal a calibrated probability, not a phrasing. Masking the loss to the target means the model learns to read, not to memorise the passages.

Chapter 3

Five kinds of example

A model that only ever sees answerable questions learns to always answer. The training set mixes five kinds of example. About 35% of them should end in a refusal. That share was chosen in the pilot sweep (15 / 25 / 35%), and the final decision threshold τ adjusts the balance afterwards.

The 20,340 training examples by kind
  • A · Answerable47.2%9,600
  • B1 · Oracle removed20.8%4,222
  • B2 · Near miss10.8%2,198
  • B3 · No / unrelated passages3.5%720
  • C · Grounded task17.7%3,600

One real example of each kind (held-out dev set)

A · Answerable — One passage states the answer; the other three are distractors (BM25 look-alikes and passages from the same document).

Question: In the "Kindness" song, what action accompanies the line "I let my Kindness Shine!"?

  1. Passage 1In the Quran, Allah commands believers to take care of orphans, recognizing their vulnerability and the responsibility the community bears towards them. The Importance of Supporting Orphaned Children Fulfilling an Islamic Duty: Supporting orphaned children is not only a noble act of kindness but also a religious obligation in Islam. Allah promises immense rewards for those who care for orphans and treat them with compassion and love. By fulfilling this duty, believers seek the pleasure of Allah and emulate the example set by Prophet Muhammad (peace be upon him).
  2. Passage 2 · the answer is hereSing the song: “Kindness” together with the moves and/or march around the room or dance: Kindness, Kindness. It feels good to be kind. (give yourself a hug; like hugging a teddy bear) Kindness, Kindness. I let my Kindness Shine! (spread hands out from heart, as if presenting) I’m kind to you. I’m kind to me.
  3. Passage 3Can be done on Random Acts of Kindness Week in February…or any time. Book Suggestions: “Kids Random Acts of Kindness” by Rosalynn Carter Blues Clues “The Sharing Surprise” by Tish Rabe 1. Have the word KINDNESS written…on a large paper or board. 2. Encourage the children to sound out each letter, and then put the sounds together to form the word. 3. Ask: What does the word Kindness mean to you?
  4. Passage 4Encourage them to think kindness wishes to themselves, their family, school, community, world. After a moment of quiet, each child is invited to place their gem in the jar. Then, talk about the fact that there are people all over the world who daily express kind wishes to them. (even people they’ve never met) Talk about how that feels to know this. Talk about nurturing our garden of Kindness. Can decorate the jar and have it on display. (as a symbol and reminder of the kindness that’s in every heart) Place a bowl/bag of gems near by.

Target (cite + quote + answer)

Source: Passage 2
Quote: "I let my Kindness Shine! (spread hands out from heart, as if presenting)"
Answer: spread hands out from heart, as if presenting

Takeaway — The hard part is the refusals that look answerable: B1 (same question, answer removed) and B2 (near misses). Easy refusals teach almost nothing.

Chapter 4

Teachers, checks & the funnel

A 125M model cannot write its own training data, so a teacher (deepseek-flash, off-peak) wrote questions, quotes and answers from passages of the training corpus. A second model family (gemini-3.8-flash) checked the dev/test labels and audited a sample of the training data.

Nothing a teacher writes is trusted as is. Every candidate is re-checked blind: the checker sees only the passages and the question, and must agree on the passage, the quote and the answer. Of 16,700 calls, 20,340 examples made it into training.

From teacher calls to training examples

Real counts from the 7.8 run. Counts can rise between stages because B1 twins and B3 items are built from accepted questions.

Teacher calls (deepseek-flash)
16,700
Questions / tasks written
25,481
Passed model-free checks
17,792
Put into 4-passage contexts
22,142
Accepted by the blind re-check
19,509
Training set
20,340

Why questions were dropped (G1, model-free checks)

6,681 of 18,084 written questions dropped before any model check.

question copies ≥ 6 passage words
3,983
vague reference (“the company”)
2,205
quote over 40 words
150
quote not verbatim
108
answer not in the quote
98

The near-miss check (V3)

“Could a careful reader give a plausible answer, including by arithmetic or from a partial statement?” It kept 2,433 of 2,835 near misses (85.8%). The rejected ones were usually answered by a distractor passage.

5% audit by a second model family

KindAuditedGemini disagrees
A4854.3%
B12150.5%
B21237.3%
C1086.5%

The stop line was 15% in any kind.

Leakage rules

  • Dev/test come only from held-out documents, scanned for 13-gram overlap with the whole pretraining corpus (< 10%).
  • 0 training passages come from a dev/test document.
  • 659 training examples were removed because they quoted a document reserved for the “memory” probe.

Takeaway — The two checks that mattered most were model-free (copied wording, vague references) and the dedicated near-miss check. The generic answerability check missed about 1 in 7 near misses that a careful reader could still answer.

Chapter 5

Training without forgetting

Fine-tuning can quietly damage what pretraining built. Two guards were set before training: held-out bits per byte on 600 untouched windows of legal and web text may rise by at most +0.02, and zero-shot CaseHOLD (legal holdings) may fall by at most 2 points.

The first pilot broke both. Measuring after every epoch showed why: forgetting peaks early, while the learning rate is high, then recovers as it decays. Raising the replay weight λ barely helped. Lowering the learning rate fixed it.

Held-out bits/byte change after each epoch (2K-example pilot, 8 epochs)

Five settings of learning rate (LR) and replay weight (λ). One epoch = 39 steps.

  • LR 2e-4, λ 0.25
  • LR 2e-4, λ 0.5
  • LR 2e-4, λ 1
  • LR 5e-5, λ 0.5
  • LR 5e-5, λ 1 (chosen)
  • Limit +0.02

For the chosen setting, CaseHOLD was measured every epoch too: -3.4, -2.3, -4.1, -5.0, -4.1, -4.8, -4.9, -4.7 points. It drops in the first epoch whatever the setting, while bits/byte barely moves.

Takeaway — The learning rate, not the replay weight, controls forgetting here. The final 20K model ends at +0.009 bits/byte (limit +0.02). CaseHOLD fell 4.8 points. A breakdown showed that drop is noisier per-item scoring rather than lost use of context, so it is reported, not gated, by the owner's decision.

Chapter 6

Decision threshold & copy decoding

Because the first token is the decision, the model's willingness to answer is one number: s = P(“none”) / (P(“none”) + P(“Pass”)). It refuses when s ≥ τ. The threshold was chosen on the dev set by a rule fixed in advance: the best balance among values of τ that refuse at least 80% of unanswerable questions and at most 20% of answerable ones. That gave τ = 0.7.

The second fix happens at generation time. Left alone, the model paraphrased inside the quote. Copy-constrained decoding allows only tokens that keep the quote a verbatim span of the cited passage, then forces Answer:. It costs nothing at training time.

Move the threshold: 499 dev questions, scored by the final model

Bars count questions per bin of the decision score s (square-root height, so small bins stay visible). Everything right of the line is refused.

87.1%

of unanswerable questions refused (target ≥ 80%)

B1 95% · B2 67% · B3 100%

1.8%

of answerable questions wrongly refused (target ≤ 20%)

Copy-constrained decoding: same model, same test set

Frozen test set, 1000 items. Plain greedy vs. copy-constrained decoding.

Quote is verbatim

Plain
39.1%
Copy-constrained
99.8%

Hallucination rate (lower is better)

Plain
53.9%
Copy-constrained
32.5%

Grounded accuracy (judged)

Plain
33.6%
Copy-constrained
31.6%

Format valid

Plain
99.8%
Copy-constrained
100.0%

Takeaway — Answer-or-refuse works well (test AUROC 0.923), and the constraint took verbatim quotes from 39% to 99.8%. It cannot make the model pick the right span: answer accuracy barely changes.

Chapter 7

More data: the learning curve

The pilot (2,000 examples, 8 epochs) cited the right passage but got only about a fifth of answers right on new questions. Checking it on questions it had trained on settled why: there it was right 95% of the time, against 23% on new ones. It had memorised its examples instead of learning to read.

So the final run used ten times more unique examples, and chose its checkpoint on answer quality, not just the answer-or-refuse decision. Each point below is scored by an independent judge model on the held-out dev set.

Learning curve (dev set, judged)

Training examples on a log scale. 2K is the pilot; 5K, 10K and 20K are the final run's curve.

  • Grounded accuracy (judged)
  • Hallucination rate
  • Grounded accuracy target 60%

Why more data: the pilot memorised

The 2,000-example pilot (8 epochs), checked on 40 questions it trained on and on the held-out dev set.

95%

Questions it trained on: right answer from the right passage

23%

New questions (dev): right answer from the right passage

Takeaway — More unique data helps steadily: about 4.1 points of grounded accuracy per doubling. At that rate, 60% would need on the order of a million examples. For a 125M model, data alone will not close the gap.

Chapter 8

Results, limits & cost

The test set is 1,000 questions from documents the model never trained on, frozen before any training, and scored once. A judge model checked each answer against the reference, and checked that the passages support it. The targets were set in advance and were not lowered afterwards.

The model met 4 of 7 targets. It missed hallucination rate, refusal recall, grounded accuracy (judged).

Frozen test set vs. targets set before training
MetricTargetSLM-125M-RAFTPlain decodingTeacher ceiling*
Hallucination rate≤ 15%32.5%missed53.9%1.0%
Refusal recall≥ 80%78.0%missed78.0%96.2%
Over-refusal≤ 20%4.8%met4.8%0.0%
Grounded accuracy (judged)≥ 60%31.6%missed33.6%100.0%
Citation accuracy≥ 80%90.3%met89.9%100.0%
Quote fidelity≥ 90%99.8%met39.1%100.0%
Format validity≥ 99%100.0%met99.8%100.0%

Also: answer-or-refuse AUROC 0.923; paired consistency 27.3% (answers correctly with the passage and refuses without it, 150 pairs). *Teacher ceiling: deepseek-flash on 200 test items. It is inflated by construction, because items were kept only where the teacher's checks agreed.

What the 32.5% hallucination rate is made of

Bold: share of the hallucinations; grey: share of all test items. Most of it is unsupported content after a citation, not answering the unanswerable.

  • Answerable, but the answer is not supported47.1%15.3% of all items
  • Grounded task output adds content32.6%10.6% of all items
  • Answered a near miss13.5%4.4% of all items
  • Answered with the passage removed6.8%2.2% of all items

Refusal recall by kind of unanswerable question

Near misses (B2) are the hard case.

Passage removed (B1)
85.3%
Near miss (B2)
56.0%
No / unrelated (B3)
100.0%

Grounded accuracy by source

Answerable test questions; judged correct and citing the right passage.

US case law
30.5%
SEC filings
37.9%
Educational web
18.8%

What it cost: ≈ $14.70 in total

Teacher APIs (DeepSeek writes and checks, Gemini audits and judges) and GPU time, by step.

Passage pools & search index (CPU)
$0.19
Teacher trial (two writers, three checkers)
$0.55
Pilot data (2.4K train, dev, test)
$1.80
Pilot sweeps & forgetting tests (GPU)
$3.58
Judging the pilot
$0.23
20K data, audit, judging the final model
$6.28
Final training & learning curve (GPU)
$2.07

Limits, plainly

  • It cites the right passage 90% of the time, but the answer is right only about a third of the time. Wrong spans and misread numbers are typical.
  • Near-miss questions are refused only 56% of the time.
  • The context is 1,024 tokens: at most 4 passages of ~160 tokens.
  • Zero-shot CaseHOLD fell 4.8 points after fine-tuning (reported, not gated).
  • It is not legal or financial advice. Always check the quote against the passage. That is what the quote is for.

Takeaway — SLM-125M-RAFT is good at deciding whether the passages answer a question, at citing the right passage and at quoting it exactly. It is weak at reading the answer out of the passage. Treat it as a citation finder whose quote you check, not as an answer machine.

Chapter 9

Beyond 125M: what scale buys

SLM-125M-RAFT cites the right passage but reads the right answer out of it only a third of the time. Is that the data, or the model? To find out, the same training rows, the same frozen test set and the same judge were used on three other models, chosen so that size and pretraining change one at a time.

  • SLM-500M (same corpus): 4× the parameters, pretrained on the same three sources (case law, SEC filings, web).
  • Gemma 3 270M: about our size outside its embeddings (100M vs 113M), but pretrained on ~6T tokens instead of 2.0B unique × 4 epochs.
  • Gemma 2 2B: bigger in both ways (2.6B parameters, ~2T tokens).

Grounded accuracy

Frozen test set, judged; target ≥ 60%. Higher is better.

SLM-125M (ours)
31.6%
SLM-500M (same corpus)
34.6%
Gemma 3 270M
64.4%
Gemma 2 2B
91.2%

Hallucination

Frozen test set, judged; target ≤ 15%. Lower is better.

SLM-125M (ours)
32.5%
SLM-500M (same corpus)
30.7%
Gemma 3 270M
20.1%
Gemma 2 2B
8.6%

Refusal recall

Frozen test set, judged; target ≥ 80%. Higher is better.

SLM-125M (ours)
78.0%
SLM-500M (same corpus)
75.0%
Gemma 3 270M
89.0%
Gemma 2 2B
90.0%
The four models side by side
ModelParameters (outside embeddings)PretrainingFine-tuningGroundedNear-miss refusalΔ bits/byteΔ CaseHOLD
SLM-125M (ours)126M (113M)2.0B unique × 4 epochs (8.2B seen)full fine-tuning + replay, 20K rows31.6%56%+0.009-4.8
SLM-500M (same corpus)518M (476M)2.2B unique × ~5 epochs (~11B seen)full fine-tuning + replay, 20K rows34.6%47%+0.018-1.5
Gemma 3 270M268M (100M)~6T tokensfull fine-tuning (forgetting gate reported, not met), 20K rows64.4%79%+0.308-9.6
Gemma 2 2B2.6B (2.0B)~2T tokensLoRA r 32, 5K rows91.2%76%+0.020-0.1

Δ bits/byte: change in the model's own held-out bits per byte on our text after fine-tuning (limit +0.02, red when over). Δ CaseHOLD: change in zero-shot legal-reasoning accuracy, in points (reported, not gated).

Reading vs. refusing

Before any fine-tuning, Gemma 2 2B (instruction-tuned, two examples in the prompt) already reads well: grounded accuracy 83.8%. But it refuses only 25% of unanswerable questions and 3% of near misses. After LoRA on 5,000 rows: refusal recall 90%, hallucination 8.6%. Fine-tuning taught it when not to answer. The reading came from pretraining.

Fair test? Checked.

  • The 500M splits validation by window, so it probably saw our test documents. Measured against text neither model saw, it showed no extra familiarity with them, and it answered under 1% of test questions with no passages.
  • Without passages, Gemma 2 answered 13.8% of test questions correctly. Excluding those, its grounded accuracy barely changes.
  • Near misses (the passages almost answer) are the weakest point for every model.
  • Copy-constrained decoding keeps small models' quotes verbatim (39% → 100% for ours). For larger models, which already quote accurately, it slightly lowers accuracy.

Takeaway — More parameters on the same 2B-token corpus barely moved the answer quality (+3.0 points). Far larger pretraining at about the same size did (+32.8 points), at a real cost in forgetting. Reading comes mostly from pretraining scale. Fine-tuning mostly teaches the model when to refuse.