Skip to main content
Every answer your engine gives is kept in the store, under its exact input. After a few thousand of them, the store holds something valuable: a labelled dataset for that exact question, in your data, with your reviews on top. hunch distill turns it into a small model that runs on your machine in about 10 milliseconds, with no API call. A small model on its own is worse than the engine that taught it. It doesn’t have to be good at everything, though. It only has to know when it is sure. Put the engine behind it, and the student answers the rows it is sure of while the engine answers the rest. hunch measures both, so you choose how much to keep local with numbers.

​
Distill

The BANKING77 spec sorts bank customers’ messages into 77 intents. Jev had answered 3,000 unlabelled messages, kept in the store. Distilling asks nothing:
Each row’s state goes through a small frozen encoder (MiniLM, on your CPU), and each question gets one linear layer, trained on your gold where a row has it and on the engine’s answer everywhere else. The rarest answer is reported because a student learns little from a handful of examples. The folder, 194 KB here, is the model; commit it with the spec. It needs the distill extra: uv add "hunch-ai[distill]", which brings fastembed and its runtime (onnxruntime, tokenizers and about 20 other packages). The encoder downloads once, about 90 MB, to ~/.cache/hunch/encoders (or FASTEMBED_CACHE_PATH).

​
Measure it like any engine

A distilled model is an engine, distilled:<folder>, so every command measures it. On the 385 held-out messages, which it never saw:
On its own it is right 91.2% of the time, where Jev is right 97.9%, and its stated confidence isn’t a trustworthy probability yet. The dial is what matters: above 0.8, it answers half the messages with no mistakes. distill holds back a fifth of any gold rows it could have learned from and records what it trained on, so test and diff grade a distilled model only on rows it never saw. (Never saw means never saw that exact text: 3 of these 385 messages differ from a training message only in case or punctuation.)

​
Put the engine behind it

intent_distilled.yml is intent.yml with the student in front and Jev behind it:
intent_distilled.yml
Jev alone, measured the same way, is estimated at 95.8% (95% CI 88.7%–97.9%); on plain accuracy the two are 97.9% and 97.7%, one message apart. Two thirds of the messages were answered on this machine. Behind a distilled model, an escalated answer always replaces the student’s: the teacher is the better model, so every doubt goes to it. Calibration fails the spec’s own test, and that is real: the rows now carry two engines’ probabilities, and the student’s aren’t honest probabilities yet. Choose act from the dial, which counts mistakes, not from the stated confidence. The share that stays local grows with the answers you keep. Students trained on more of Jev’s answers, measured on the same holdout with a script of the same design (scikit-learn in place of hunch’s own linear layer, which matches it; see the design notes, round 17), kept more: Every run and every review adds to the store, so distilling again later keeps more of the work local.

​
When not to use it

A student learns an answer from examples of it. When one answer is rare and missing it is costly, a few thousand rows aren’t enough. The command guard is that case: 2% of shell commands need a person, so its student saw only 9 destructive commands. Measured by a script on commands the panel had labelled and the student never saw, the student with Jev behind it (act 0.9) let through 4 of 8 destructive commands and 6 of 29 that send data out, where Jev alone let through none and one. distill flags the rare answers; hunch test --model distilled:… shows the misses on the dial’s “act on no” side. For a guard like that, keep the engine until the store holds many more of the rare answer. A distilled model answers only the questions it was trained on, word for word. Change a question and it refuses until you distill again, and hunch diff against the old engine shows what the new student changed.

​
Speed and cost

Inside a running app, a new answer takes about 11 ms: the encoder and the linear layer, on the CPU. A fresh process pays about 0.3 s to load the encoder first, so a per-command hook gains no speed, only cost and network. Distilled answers cost nothing and are stored like any other, so a row seen before comes from the store.