Skip to content

Vaultr Journal · AI & the law

Small models, private data: why the biggest model is not the answer 

Frontier models impress on general knowledge. Legal work needs something narrower — reading discipline on your corpus, on your machine. On the economics and honesty of small local models.

6 September 2026 · 5 min read

Back to Journal

The largest models make the best headlines and the worst procurement decisions for confidential work. The reason is not capability — frontier systems are extraordinary — but where the computation happens and what it costs in control.

The legal task is narrower than the general task

Reading a matter is not answering humanity's questions. It is: retrieve the relevant passages, reason within them, cite precisely, decline when they are absent. That task fits mid-sized open-weight models — the 7–14 billion parameter class that runs on a well-specced laptop — far better than the raw benchmarks suggest, because the grounding discipline does most of the work. Retrieval selects; the model reads carefully; the citation enforces honesty.

What you trade, stated plainly

  • Breadth: a small local model knows less of the world. For matter-grounded work that is nearly irrelevant — and occasionally a virtue.
  • Depth of reasoning on genuinely novel questions: real, and the reason external endpoints remain an explicit, named mode rather than a banned one.
  • Hardware: the tested floor is 16 GB of unified memory; long bundles want more. That is a stated requirement, not a hidden one.

The prize justifies the trade: prompts and passages that never leave the device, no per-token metering of your matters, and a model choice you can change per matter. When a task genuinely needs more capability, the external mode exists — named, shown before the first run, and different on purpose. The architecture makes the upgrade a decision instead of a default, which is exactly where a decision about disclosure belongs.

Pick the smallest model that holds up for the task, and keep it inside the boundary. Escalate on purpose, never by accident.