Skip to content
SLM-125M

Model arena · grounded QA

Same passages, same question, every model

Each model gets up to four passages and a question. It must cite one passage, quote it word for word and answer, or say “The passages do not contain the answer.” Every answer was scored by the same rules and the same Gemini Flash judge, and every fine-tuned model learned from the same dataset. Pick an item to see what each model said and how it scored. Then see the leaderboard, what scale buys, and what each model cost.

  • 7

    models

    4 bases, RAFT-tuned, + 2 DPO + 1 untuned

  • 1,499

    items scored

    frozen test 1,000 + dev 499

  • 833

    items you can browse

    SEC filings and web text

  • 60 pts

    Gemma 2 2B over ours

    grounded accuracy, test

These are the saved answers from the evaluation runs. Nothing is generated here. To ask SLM-125M-RAFT your own question live, use the Grounded QA demo. Court opinions name private people, so case-law items count in the leaderboard but are not shown, and other items are withheld if an automatic privacy check flags them.

Browse the answers

Filter by kind of question, jump to items where the models disagree, or search. Use ← → to step through. Each answer gets one outcome: correct, a correct refusal, or what went wrong.

549 of 1,000 items shown. Never used for any choice.

Loading judged items…

Leaderboard

Every item, including those not shown above. These are the numbers reported in the project docs, reproduced item by item when this page was built.

Leaderboard

Every item in the set, judged. Click a column to sort. ✓ = meets the target set before training; best in each column in bold.

Model
Target≥ 60%≤ 15%≥ 80%—≤ 20%≥ 80%≥ 90%
Gemma 2 2B-RAFT (LoRA)2.6B · ~2T tokens (Google) · LoRA r 32, 5K rows91.2%✓8.6%✓90.0%✓76%0.6%✓98.0%✓99.1%✓
Gemma 2 2B-it, untuned2.6B · ~2T tokens + Google's instruction tuning · none (2-shot prompt)83.8%✓34.3%·24.7%·3%0.0%✓90.4%✓100.0%✓
Gemma 3 270M-RAFT268M · ~6T tokens (Google) · full FT, 20K rows64.4%✓20.1%·89.0%✓79%3.4%✓97.3%✓98.2%✓
SLM-500M-RAFT518M · ~11B tokens, same 3 sources · full FT, 20K rows34.6%·30.7%·75.0%·47%4.6%✓89.5%✓99.8%✓
SLM-125M-RAFT126M · 8.2B tokens (ours) · full FT, 20K rows31.6%·32.5%·78.0%·56%4.8%✓90.3%✓99.8%✓

Test: 1,000 items (500 answerable, 300 to refuse, 200 tasks), never used for any choice. The DPO models have not been run on it yet.

What scale buys

4× the parameters on the same three sources (SLM-500M) moved grounded accuracy by 3 points, which is within noise. About our size, pretrained on trillions of tokens (Gemma 3 270M): 33 points. Fine-tuning mostly teaches when to refuse: untuned Gemma 2 2B-it already reads well (84%) but refuses only 25% of unanswerable questions.

Reading comes from pretraining scale

Grounded accuracy on the frozen test set against tokens seen in pretraining (log scale). Same RAFT data, same judge.

  • Case law, SEC filings, web (~2B unique tokens)
  • Google's pretraining

Reading the comparison fairly

  • The judge (Gemini Flash) compares each answer with a reference answer. It is a strong check, not ground truth.
  • One training run per model. On the test set, differences under about 5 points in grounded accuracy, or 10 in near-miss refusal, are within sampling noise.
  • SLM-500M may have seen some test documents in pretraining. A check against unseen text found no sign that it helped.
  • Gemma 2 2B was tuned with LoRA on 5,000 rows; the others had full fine-tuning on 20,000.
  • Gemma 3 270M forgot more of our text than the limit allowed (held-out bits/byte +0.31 vs +0.02). It reads better, but it models legal and financial text less well afterwards.
  • RLAIF → DPO (round 1) is on dev only. Hallucination fell 1.6 points on both models, short of the 5-point bar set beforehand and within noise. The only clear change is that Gemma 3 refused 3.6 points more answerable questions.
  • Near misses (a passage on the topic that lacks the detail asked) are the weak point for every model.

What it cost

Pretraining is the expensive step, and only one of these base models was pretrained by us. Once a strong base exists, teaching it the task costs a few dollars.

What each model cost us to build

Our own spend (Modal GPUs + LLM APIs), from the phase ledgers, on one axis. Includes every pilot and sweep, not just the final run.

  • Pretraining (ours)
  • RAFT dataset + judging (shared)
  • Fine-tuning GPU (incl. sweeps)
  • Judging / AI labels
  • SLM-125M-RAFT
    $40.03

    pretraining ≤ $25.32 (8 × H100, 45 min, incl. calibration); RAFT dataset + judging (DeepSeek, Gemini) $9.06, shared by every model below; SFT sweeps + final run $5.65

  • SLM-500M-RAFT
    $5.46

    pretrained by its author (thesreedath), not by us; our cost: fine-tuning $4.74 + judging $0.72

  • Gemma 3 270M-RAFT
    $4.75

    pretrained by Google (~6T tokens); our cost: fine-tuning (full FT + LoRA pilots) $3.86 + judging $0.89

  • Gemma 2 2B-RAFT
    $8.70

    pretrained by Google (~2T tokens); our cost: baselines, LoRA sweep and test $7.51 + judging $1.19

  • RLAIF → DPO, round 1 (both models)
    $7.45

    sampling, headroom check and DPO on SLM-125M and Gemma 3 270M $2.56; Gemini labels + judging $4.89

Pretraining our own 125M model was the biggest single bill ($25.32), and it bought a model that reads 32% of answers correctly. Google's Gemma 3 270M, pretrained on ~750× more text at no cost to us, reached 64% on the same dataset for $4.75 more, though it forgot far more of our text. Everything on this page cost about $66.39.