From public data to a working legal language model

The obvious recipe for a legal model is mostly legal text: say 70% court opinions, 20% SEC filings, 10% web, about 10 billion tokens in all. Before downloading anything we measured how much clean text each source actually holds, by cleaning 2,000 sample documents per source and projecting to the full dataset.

The case-law dataset has only 282,390 documents. After cleaning that is about 0.81B tokens, while 70% of 10B would need 7.00B: a 8.6× shortfall. The recipe was impossible.

The three datasets live on Hugging Face as parquet shards. We never download them. Each shard gets its own container that streams rows, cleans each document on the fly, writes the survivors, and stops at its token cap. In total, 718,780 documents were streamed out of more than 10.0M available.

Why one container per shard? Cloud machines get preempted, and a preempted job restarts from zero. With one worker per shard, a preemption costs one shard, not the whole run. That happened for real: twice during tokenization, and only the affected shard re-ran.

Raw text is noisy: court opinions are often scanned paper with OCR errors, and filings are full of headers and signature blocks. Every document passes through the same six fixed rules, cheapest check first. Every rule is deterministic, so the same input always gives the same corpus, and every drop is counted by reason.

Of 718,780 documents streamed, 670,124 made it into the final corpus (93.2%).

Courts reuse templates, and the same filing can appear twice. Duplicates waste training budget and invite memorization. Exact copies are easy: hash the normalised text. Near copies, the same order with a different date, need MinHash.

Each document becomes a set of overlapping 5-word shingles. Two documents are near-duplicates when their sets overlap heavily: Jaccard similarity of at least 0.8. Comparing every pair of 232,292 opinions is 27 billion comparisons. MinHash compresses each document to 32 numbers whose agreement estimates the Jaccard similarity, and LSH buckets them so only likely pairs are ever compared.

We want to test the model honestly on CaseHOLD, a legal benchmark built from court opinions. If its test passages also appear in the training text, a good score would just mean memorized, not learned.

So we collected every 13-word window of the benchmark (480,908 of them) and dropped any training document that shares even one. Thirteen words is long enough that a shared window almost never happens by chance.

Models read tokens, not characters. We trained a fresh byte-level BPE tokenizer on this corpus: it starts from the 256 possible bytes and repeatedly merges the most frequent adjacent pair, until it has 16,384 tokens. Because it was trained on legal and financial text, words like “plaintiff” and “pursuant” became single tokens.

Byte-level means there is no unknown token, ever. Any string, including OCR garbage, emoji or§, can be encoded.

Training needs fixed-size inputs. Every document is tokenized, followed by an <|eos|>, and the whole stream is cut into 1,024-token windows, with no padding wasted. Each window is stored as raw 16-bit integers: a 16,384-token vocabulary fits in 16 bits, half the disk of 32-bit.

1% must be held out for validation, and how you choose it matters. Our first version followed the original spec: every 100th window. But documents span windows. An SEC filing averages 18.8 windows, so a validation window's other ~18 windows sat in training. After 4 epochs over those siblings, “held-out” loss would partly measure memory.

A Llama-style decoder: 12 identical layers of attention and MLP, 768-wide, with 12 heads, RMSNorm, rotary positions (RoPE), SwiGLU and tied input/output embeddings. These are the same ingredients as much larger open models, so everything here scales up.

Exactly 125,848,320 parameters, all trainable. Click a block to see what it does.

15,564 optimizer steps of 524,288 tokens: 8.16B tokens, four passes over the corpus, on 8 × H100 in 44.9 minutes. The learning rate follows one continuous schedule: 381 warmup steps to 6e-4, then a single cosine decay to 6e-5, never restarted, even across epochs.

Repeating data risks memorization, so the held-out curve (documents the model never sees) is the one that matters. It improved at every one of its 16 evaluations.

On held-out documents the model reaches 0.657 bits per byte (loss 2.127, perplexity 8.39). Bits per byte is the honest headline: it doesn't depend on the tokenizer, so it compares across models. MMLU-style benchmarks are near-random at this size.

The split by source shows what it learned: formulaic SEC language best, case law well, general web text worst.