A row is one thing you judge
A row is one shell command, one agent turn, or one product review, with an id named by the spec’skey. Rows come from a CSV file, coding-agent session logs (traces(...)), or a Python function (py(file.py:fn)).
A question asks one thing of every row
A question hasinstructions, which say what is being asked, and usually criteria, which describe the possible answers. The model sees these together with the row’s state, the columns you chose to show it. It never sees the question’s name, so the instructions must carry the whole meaning.
The answer is always typed, never free text. The type decides its shape:
A spec writes the questions down
A spec is a YAML file for one judgment. It says where the rows come from, what the model sees, which questions to ask, and which tests the answers must pass. It lives in git beside your code. A folder of specs is a project, where one judgment can read another’s results.Every answer is stored under its exact input
When hunch asks a question, it stores the answer in.hunch/store.sqlite. The answer is filed under a key made from the exact request: the model, the state, and the question as sent. The same request is never paid for twice.
Here is what goes into that key, and what stays out:
- A re-run with nothing changed costs nothing.
- Changing one question asks only that question again.
- Old answers stay in the store, so
diffcan compare two versions of a spec without asking anything.
Gold is the answer you trust
To say how often the model is right, you need rows where the right answer is known. hunch calls that gold. It comes from an answer-key column in your data (gold:), or from your own verdicts in review.
An answer key is someone else’s judgment, and it can be wrong (one public key was wrong four times as often as the model). So hunch treats a key as a starting point: the rows where the model and the key disagree are the first ones it asks you about.
act decides what a person sees
An answer given at 0.55 deserves less trust than one given at 0.99.act is the confidence below which an answer is not acted on automatically; those rows go to review instead.
Choosing act is a trade. Set it higher and fewer rows are automated, but fewer of those are wrong. hunch test shows this trade as a dial, with one line per threshold, so you can pick with numbers. Neither act nor gold is sent to the model, so changing them asks nothing.
Here is that trade on 770 real answers from the banking cookbook. Each dot is one message, the most confident at the top. Move act and watch mistakes leave the automated block and land with a person:
At 0.90, 82.6% of answers are automated and 8.2% of those are wrong. Point at a red dot near the top: some of the model’s most confident “mistakes” are the answer key’s.
Review turns your verdicts into gold
hunch review shows you rows one at a time and records your verdict. It orders them by how much each verdict tells hunch:
- rows where the model and the answer key disagree
- a random sample of the rest
- rows below
act
test estimate accuracy honestly, with a range around it.
A verdict belongs to the exact text you judged. If the row’s text changes, the old verdict no longer counts, and the row comes back to the queue.
An engine answers the questions
The examples use TypeSafe’s Jev, a model built to return typed answers with probabilities. Any LLM that exposes token probabilities also works. See Engines.How the pieces meet
Work goes round this loop:run a spec, review a sample, test, then change the spec and diff to see what flipped.
Your application can call the same spec with hunch.judge(...).