Skip to main content
A git diff of a spec shows the words you changed. It doesn’t show what those words do. Narrowing one question by a clause can move dozens of answers, and a wording that reads better can be worse.

​
diff shows what flipped

hunch diff runs two versions of a spec on the same rows and compares them row by row. Here is the command guard on 1,315 shell commands real coding agents ran. The candidate narrows sends_out from “send code, files or data to another machine” to “publish or upload something where other people or services can see it”:
20 commands changed answer. Against the answer key, a review panel’s verdicts, 2 were fixed and 18 broken, and a sign test says that balance is not chance. Each flip is listed, ✓ for fixed and ✗ for broken, so you can read the ones that went wrong. A flip marked ~noise was a near tie that could tip either way on a re-run. Reading them is the point. Most of the broken rows install an app on a phone over adb. The panel counted that as sending files to another machine, and it is; the narrower wording says a test phone isn’t “where other people can see it”. So the diff isn’t only saying the new wording is worse. It is saying the new wording is a different policy, and showing exactly which commands the policy changes. Whether adb install should wait for a person is your call; the diff makes sure you make it on purpose. --against takes another spec file, a folder, or git:REF for the version committed at that ref:
The old version always runs on today’s rows, so the comparison is fair. Its answers are usually already in the store, and a question whose text didn’t change reports unchanged (same keys, 0 calls), as destroys and reaches_outside did above. Every line of the output is explained in CLI › diff. A changed question has to be asked again on every row: this one cost $0.03 for 1,315 commands. To see what a change does before paying for all of them, add --sample 200: both versions are judged on the same 200 rows, and those answers are kept for the full run.

​
Only the flips carry evidence

Where both versions give the same answer, they are right or wrong together, so those rows can’t tell you which is better. The flips can. When there is no answer key, or the key is thin, review just those:
The queue starts with the rows where the two versions disagree, marking one answer ← --against and the other ← this spec. Your verdicts become gold, so the next diff can count fixed and broken.

​
Apps depend on answer names

diff says whether the new answers are better. It can’t see the code that acts on them. Take the outcome judgment, which reads the developer’s next message to say how a coding agent’s turn went: worked, failed, redirected or unclear. A job that re-runs failed turns does something like this:
That line depends on the answer’s name. Rename failed to broken in the spec and the model answers broken from then on. The branch never matches again, and nothing in hunch knows the job was looking for the old name. If the answer key was renamed along with it, every test still passes. The spec can say which answers the job relies on. Name it under exposures, and map each question it reads to the answers it compares against:
outcome.yml
Every command lints the spec before it does anything, so after the rename lint, test and diff all stop, in CI too:
The rename is still possible. It now takes a second edit to uses, made by someone who has changed the job, and the owner is right there in the spec. The same list makes diff precise. The Claude Code cookbook tried telling the model that a follow-up question is not a failure. On the 309 public turns, diff ends with the rows each exposure would see change:
19 turns changed answer, but only 7 moved into or out of failed, so only 7 would change what the retry job does. The dashboard shows every answer, so all 19 reach it. The values each question type accepts are in the spec reference.

​
Trying a candidate on live traffic

Sometimes the rows that matter are the ones arriving in production, not a CSV. You can run a candidate spec next to the live one, unseen by users. Here the live folder holds the quickstart’s command guard and the candidate folder the same guard with the narrower sends_out, both under one project folder so they share its store. Each command the agent is about to run goes through:
app.py
The live spec answers and judge returns that answer. The row is logged to the store’s traffic table, redacted by the live spec’s redact rules. The candidate then answers the same row in the background. It never delays or changes the live answer, and if it fails, the error goes to stderr. After 38 commands have come through, --traffic compares the two on the logged rows. Both sets of answers are already in the store, so this costs nothing:
Logged traffic has no answer key. hunch review candidate --against live --traffic queues the flipped rows, and your verdicts become the gold that the next diff scores against. A candidate doesn’t have to exist before the traffic does. Call judge(LIVE, log=True, ...) in production, and any spec written later can be compared on those rows. Through the server, pass "shadow" in the body of POST /v1/judge.

​
What happens to old answers

An answer is stored under a key made from the engine, the row, and the question as sent. Edit the question and every row gets a new key. What run does next is set per judgment with on_change: Under freeze, run stops and says so:
Check with diff, then accept with hunch run --allow-change. test, diff, review and judge() always use the spec as written, so they show what an edit would do whatever on_change says.

​
Letting a model propose rewrites

hunch suggest asks an LLM to rewrite one question’s wording, and keeps a rewrite only if it beats the current wording on gold rows the writer never saw:
Rewrites are saved under .hunch/suggest/. Treat a kept one as a candidate: the best of several looks better than it is, so diff it on rows that played no part in choosing it before you adopt it. How the numbers work explains why, and Route banking intents shows what it gained on BANKING77.