> ## Documentation Index > Fetch the complete documentation index at: https://fuguai.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # CLI > Every hunch command and flag, what each line of output means, and the exit codes. ```bash theme={null} hunch [flags] hunch init [--list] RECIPE [DIR] ``` `path` is a spec file or a folder of specs. Every command lints the whole project first: warnings go to stderr; an error exits 2. ## Commands | Command | What it does | Asks the engine? | | --------- | -------------------------------------------------------------------------------------------- | ---------------------------- | | `lint` | Checks the specs and prints `lint: ok (N judgments)` | No | | `compile` | Shows one real request and what `run` would ask and cost | No | | `run` | Answers every row, caches the answers, writes one table per judgment | Yes, missing answers only | | `test` | Scores the answers against gold and checks the `tests:` thresholds | Yes, missing answers only | | `review` | Walks you through the rows that need a human verdict | Yes, missing answers only | | `diff` | Compares this spec with an older or other version on today's rows | Yes, missing answers only | | `suggest` | Asks an LLM to rewrite one question and keeps a rewrite only if it wins | Yes (writer and new answers) | | `init` | Copies a starter recipe into a folder | No | | `skill` | Installs the skill that teaches a coding agent to use hunch | No | | `hook` | Guards a coding agent's shell commands: `install` wires it in, `run` is what the agent calls | Yes, per command | | `docs` | Writes a page people can read and search: each judgment's status, numbers and lineage | No | Commands ask only for answers the store lacks. `--max-cost 0` guarantees no spend: if anything is missing, the command stops before asking. For `suggest`, the cap also covers each writer call at its worst case. ### `hunch init` ```bash theme={null} hunch init --list # the recipes, one line each hunch init agent-commands # copies the recipe into ./agent-commands hunch init agent-eval evals/ # copies into evals/ ``` The folder must not exist. Recipes: `agent-commands` (a guard for 38 real coding-agent shell commands, the [quickstart](/quickstart), about \$0.002), `agent-eval` (check a coding agent's work from its trace), `tickets` (40 made-up support tickets: a choice, a yes/no and a score). ### `hunch skill` ```bash theme={null} hunch skill # into this folder hunch skill ~/ # for every project on this machine ``` Writes the hunch skill to `.claude/skills/hunch/SKILL.md` (Claude Code) and `.agents/skills/hunch/SKILL.md` (Codex; Cursor reads both). It tells the agent how to run the loop (`lint`, `compile`, `run` with a cost cap, `test`, `diff`) and what not to do: spend without `--max-cost`, write `*.reviews.csv`, or edit `.hunch/`. Each file prints `installed`, `updated` or `up to date`; an existing copy is replaced, so re-run after upgrading hunch and commit the result. ### `hunch hook` ```bash theme={null} hunch hook install # Claude Code and Codex, in this folder hunch hook install --agent claude # one agent: claude, codex or both hunch hook run hunch/command_guard.yml # what the agent calls, with the event on stdin (--deny for Codex) ``` `install` copies the command guard (spec, 38 labelled commands, NOTICE) into `hunch/`, keeping files already there, adds `.hunch/` to `.gitignore`, and adds a `PreToolUse` hook with matcher `Bash` and a 30-second timeout to `.claude/settings.json` and `.codex/hooks.json`, keeping the rest of each file. It writes nothing if a settings file can't be read. The hook command finds the spec through `$CLAUDE_PROJECT_DIR` (Claude Code) or the git root (Codex), and ends in `|| echo` a warning, so a missing or older hunch lets commands through instead of blocking them. `run` reads the agent's event, judges the command with `request` (the last message the person typed, from a Claude Code transcript; empty otherwise), `cwd`, `description` and `command`, and logs the row, redacted by the spec's rules, for `--traffic`. When any answer is `yes` or routes to `review`, it replies `ask`, or `deny` with `--deny` (Codex, which cannot ask yet), with the reasons; otherwise it prints nothing. It never replies `allow`. On any error it prints a `systemMessage` warning and lets the command through. `HUNCH_MAX_COST` defaults to 0.01 per command. See [Guard your coding agent](/guides/guard-your-agent). ### `hunch docs` ```bash theme={null} hunch docs support/ ``` Writes two files beside the results file (see below), named after the spec or folder: `.html`, one self-contained page with no server or network needed, and `.manifest.json`. It reads the specs, the results of the last `test` of the same path and the store's run log, and never asks the engine. See [Share what your judgments decide](/guides/share). #### The manifest | Field | Meaning | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `version` | Format version (1). Field names are stable within a version. | | `git_sha`, `project` | The commit, and the spec or folder name. No timestamp, so an unchanged project writes an identical file. | | `judgments.` | The spec as written, every key it sets (a `multi` question stays one question), plus `file` (the spec's path), `spec_hash` (the hash `test` records; the page marks a judgment stale when they differ) and `upstream` (the judgments it reads). | ## Flags | Flag | Used by | Default | What it does | | ----------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--version` | alone | | Prints the installed version and exits. | | `--node NAME` | `run`, `test`, `diff`, `review`, `compile`, `suggest` | all (`run`, `test`), required when a project has several judgments (`review`, `suggest`) | One judgment in a project. `run --node X` runs X and the judgments it reads from. | | `--against REF` | `diff`, `review` | none | `diff`: the old version, a spec/folder path or `git:REF` (for example `git:HEAD~1`); with `--model` and no `--against`, the same specs on their own engine. `review`: put rows where that version answers differently first | | `--source CSV` | all | the spec's `source` | Run the root judgments on this CSV instead (a holdout set, for example) | | `--traffic` | all | off | Run the root judgments on rows logged by `judge(..., shadow=...)`. Not with `--source` | | `--model ID` | all | the spec's `model` | Use this engine for every judgment. Its tables get a suffix (`intent__deepseek_deepseek_flash`), so the spec's own tables are untouched | | `--max-cost USD` | all | `$HUNCH_MAX_COST`, else no cap | Stop before asking if one judgment's missing answers would cost more than this (estimated) | | `--sample N` | `compile`, `run`, `test`, `diff` | all rows | Judge only N rows of each root source, the same N every time | | `--allow-change` | `run` | off | Accept a changed spec under `on_change: freeze` | | `--list` | `review` | off | Print the queue instead of prompting | | `--limit N` | `review` | no limit | At most N rows this session | | `--audit N` | `review` | 30 | Random rows to spot check per question, counting ones already reviewed | | `--reviewer NAME` | `review` | your login name | Name saved with each verdict | | `--question QID` | `suggest` | required if the spec has several questions | The question to rewrite | | `--n N` | `suggest` | 3 | Rewrites to try | | `--writer MODEL` | `suggest` | `deepseek:deepseek-flash` | The LLM that writes rewrites | `--sample` picks rows by a hash of the key, so a repeated sample comes from the store and a later full run asks only the rest. `run --sample` caches its answers but does not replace any table, since downstream readers would take the sample for the whole. ## Exit codes | Code | Meaning | | ---- | ----------------------------------------------------------------------------------------------------------------------------------------- | | 0 | Done. For `test`: every check passed | | 1 | `test`: at least one `FAIL` line. Any command: stopped with a message (above `--max-cost`, `on_change: freeze` refused, unknown judgment) | | 2 | Lint error in a spec, or bad arguments | `test` exits 1 both when a check fails and when `--max-cost` stops it from asking (for example, `order_stability` needs new answers). Read the last line of output to tell them apart. ## Reading the output ### `compile` ```text theme={null} # intent: request for row 1 (one request per row, all questions read once) { "model": "jev-1.13.0", "state": { "text": "I still have not received my new card, I ordered over a week ago." }, "questions": { "intent": { "type": "choice", ... } } } # intent: 770 rows in; 770 answers planned, 770 cached, 0 to ask, ~0 input tokens, ~$0.00000 (jev-1.13.0) ``` | Line | Meaning | | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | JSON | The exact request for the first row, after `redact` and `clip`. For an LLM engine, the prompt. | | `# : …` | Rows in, answers planned, cached, to ask, estimated tokens and cost. | | `where keeps ≤ N`, `rows kept because the answers their where-clause needs aren't cached yet` | Upstream hasn't run; count and cost are upper bounds. | | `size warning` | A row's state is near the engine's limit. | | `on_change: freeze and the spec changed` | `run` will refuse without `--allow-change`. | | `# total` | The whole project, with an expected cost when some rows are upper bounds. | ### `run` ```text theme={null} intent: 385 rows in; answers: 385 cached, 0 asked in 0 requests, 0 input tokens, $0.00000 intent: 74 rows below act=0.9 → review queue materialized 1 table(s) in .hunch/store.sqlite (run 2026-09-25T10:45:03-8b2bab) ``` * First line: answers cached and asked, requests, input tokens, actual cost. * `rows below act`: confidence under `act`; `review` shows them as "model unsure". * The run id is stored on every table row ([Store](/reference/store)). * stderr shows progress while asking: `asking 529 answers in 529 requests (~$0.0168)` and the request rate. ### `test` Real output, BANKING77 holdout (`hunch test intent.yml --source banking77_holdout.csv`), trimmed: ```text theme={null} intent (choice, 385 rows) gold: 385 rows (243 from source, 142 from review, 25 with two acceptable labels) PASS estimated accuracy 95.8% (95% CI 88.7%–97.9%) from reviews of 60/340 agreeing rows, 45/45 disagreeing rows (min 85%) not the headline: on current gold 97.9% (trusts unreviewed rows), on the raw answer key 88.3% PASS calibration error 0.043 (raw source gold: 0.053) (max 0.05) stated p n avg stated observed 0.4–0.5 4 0.450 0.750 ... 0.9–1.0 311 0.990 1.000 dial automated wrong among automated 0.50 99.0% 1.8% ... 0.90 80.8% 0.0% ← act PASS accuracy among auto-acted 100.0% on 81% of rows at act=0.9 (min 97%) most confident mistakes (8 total; high confidence + wrong = dangerous, or a gold error → hunch review): # 560 gold=card_acceptance got contactless_not_working 0.89 Are contactless payments enabled on my new card? most confused (gold → got): 1 card_acceptance → contactless_not_working PASS order stability: 11/300 answers flip (3.7%, 8 within noise band), mean |Δp| of original answer 0.032 (max flip rate 5%) answers: 685 cached, 0 asked in 0 requests, 0 input tokens, $0.00000 ``` | Line | Meaning | | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gold:` | Rows with a right answer, by origin: source column, reviews, two acceptable labels. Ambiguous and needs-context rows are excluded. | | `context:` | After "needs more context" verdicts: the share of random spot checks that needed more than the state shows. | | `estimated accuracy` | The headline, from reviewed rows, with a 95% interval. With an answer key: all disagreements plus a random sample of agreements, weighted to all rows. Without: random spot checks only. `(min …)` is the threshold. See [How the numbers work](/reference/how-the-numbers-work). | | `accuracy` | Shown instead when there is no estimate yet: plain accuracy on current gold. `upper bound only, no estimate` means spot checks are missing. | | `not the headline` | Accuracy on current gold (trusts unreviewed rows) and on the raw answer key. | | `calibration error` | Row-weighted gap between `stated p` and `observed`, with the per-band table. Observed above stated: too modest; below: overconfident. | | `AUROC` | Yes/no: how well p(yes) separates yes from no. 0.5 is chance, 1.0 perfect. | | `dial` | Per threshold: share of rows automated and share of those wrong. `← act` marks `act`; yes/no questions get one column pair per side. Computed on rows with gold. | | `accuracy among auto-acted` | Accuracy on rows at or above `act`. | | `most confident mistakes` | Wrong answers, most confident first: a model error or a gold error. | | `most confused` | Most common (gold → got) pairs. | | `order stability` | With the `order_stability` test: flips when options are shuffled; noise-band flips (margin under 0.10) are expected. | | `declined (none_of_these)`, `escalated to …` | Questions with `none` or `escalate`. | | `weighted to the population` | The source has `weights`; every number is weighted. | | last line | Answers cached and asked, and cost. | #### The results file When `test` finishes, it also writes what it printed as JSON, for CI, dashboards and agents: `.hunch/target/.json`, beside the store, with the path of the tested spec or folder relative to the store's folder (`hunch test examples/tickets/triage.yml` writes `.hunch/target/examples/tickets/triage.json`). With `--model`, the engine's suffix is added, as for its tables (`triage__deepseek_deepseek_flash.json`), so another engine's results never replace the spec's own. `test` removes the file when it starts, so a run stopped by `--max-cost` leaves none rather than an old one. | Field | Meaning | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `version` | Format version (1). Field names are stable within a version. | | `command`, `at`, `git_sha` | `test`, when it ran (UTC), the commit it ran on. | | `passed` | Every check passed, or failed only with `severity: warn` (the exit code was 0). | | `sample` | N if `test` ran with `--sample N`, else `null`. | | `cost` | Spent by this run, in USD. | | `judgments.` | `spec_hash`, `model`, `rows`, and `questions` (and `multi`, for `type: multi`). | | `questions..gold` | Rows with gold: `rows`, and by origin `source`, `review`, `excluded`. | | `questions..accuracy` | `value`, `ci` (`[low, high]`, or `null` when `basis` is `gold`: plain accuracy against gold, no estimate), `basis` (`estimate` or `gold`), `reviewed` (per group: `reviewed` of `of` rows). | | `questions..calibration_error`, `auroc` | As printed; `auroc` for yes/no questions with both answers in gold. | | `questions..act` | With `act` in the spec: `threshold`, `automated` (share of all rows acted on), `judged` (acted-on rows with gold), `wrong` (share of those that are wrong; `null` if none has gold). | | `questions..dial` | Per threshold: `automated` and `wrong` (shares); yes/no questions have them under `yes` and `no`. | | `questions..mistakes` | `total`, and `most_confident`: `id`, `got`, `p`, `gold`. | | `questions..checks` | Each check: `check` (the test key, like `min_accuracy`), `passed`, `value`, `limit`, `severity`. | | `questions..spot_checks`, `needs_context` | Random spot checks, and how many of them needed more context. | | `judgments..metrics.` | `rule`, `rows`, `fired`, `rate`; with gold, `gold` (`basis`, `rows`), and `gold_rate`, `missed`, `false_alarms` (each `count`, `of`, `rate`, `ci`); `checks`. See [metrics](/reference/spec#metrics). | | `judgments..examples` | Per example: `name`, `passed`, and `answers` (`question`, `expected`, `got`, `p`, `passed`). See [examples](/reference/spec#examples). | ### `diff` Real output, BANKING77 (`hunch diff intent.yml --against intent_bare.yml`), trimmed: ```text theme={null} answers: 1540 cached, 0 asked in 0 requests, 0 input tokens, $0.00000 ══ intent vs intent_bare intent: 77/770 rows flip (15 within noise band) probabilities moved: mean |Δ| 0.102, mean Δ p(old answer) -0.021 gold accuracy on the 770 shared rows with gold 79.0% → 85.5% (✓ 58 fixed, ✗ 8 broken, 11 wrong both times, 0 without gold) paired sign test p=0.000 → significant ✓ # 76 activate_my_card 0.92 → card_linking 0.84 Could you help me reactivate my card? I… ✗ # 255 fiat_currency_support 0.66 → exchange_via_app 0.91 How do I change currencies to euros? # 268 exchange_rate 0.68 → exchange_via_app 0.94 I want to make a currency exchange to E… … 65 more ``` | Line | Meaning | | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `rows flip` | Rows whose answer changed; `within noise band` counts flips where either side's margin is under 0.10. | | `N rows newly reach it, M no longer do` | An upstream `where` changed which rows this judgment sees. | | `probabilities moved` | `mean \|Δ\|`: size of the move. `mean Δ p(old answer)`: direction (for yes/no, the change in p(yes)). | | `gold accuracy … →` | Old and new accuracy on shared rows with gold: fixed, broken, wrong both times, without gold. | | `paired sign test` | Whether fixed vs broken beats chance. | | flip rows | ✓ fixed, ✗ broken, blank: no gold or wrong both times. `~noise` flips are listed last. | | `unchanged (same keys, 0 calls)`, `new question`, `renamed from 'x'`, `moved by upstream changes` | Other outcomes; renames are matched by answer keys. | | `affects …: N rows` | Per [exposure](/reference/spec#exposures), the rows whose change it can see: an answer changed, appeared or disappeared on a question it reads, and, if its `uses` names the [answers it relies on](/reference/spec#answers-an-exposure-relies-on), one of those was the old or the new answer. | | `not affected: …` | Exposures that read a question in the diff but see no change. | Both sides use today's reviews. ### `review` See [Review](/guides/review) for the screen and keys. `--list` prints one row per item: ```text theme={null} review queue: 112 disputed, 30 audit disputed # 281 intent card_arrival 1.00 gold=card_delivery_estimate my card was not in the mail again can you advise? ``` Columns: kind, row id, question, model answer and confidence, answer key (`-` if none), start of the row. Verdicts saved to `.reviews.csv` (columns `qid,row_id,state_hash,verdict,label,reviewer,at,kind`): | Verdict | Saved when | | ------------------------------ | ---------------------------------------------------------------------------------------------------- | | `model_right` / `key_right` | On a disputed row, you picked the model's answer / the answer key | | `against_right` / `spec_right` | On a shadow row, you picked the `--against` spec's answer / this spec's | | `confirmed` | On a spot check, you picked the marked answer | | `labeled` | You picked any other option | | `both_ok` | `b`: two answers are acceptable; both count as right | | `ambiguous` | `a`: no single right answer; left out of scoring | | `needs_context` | `c`: the text shown is not enough to judge; left out of scoring and counted in `test`'s context line | `kind` says why the row was reviewed: `disputed`, `audit` (a random spot check), `uncertain`, `shadow`, or any other name you write when importing verdicts (for example `flagged`). Only `audit` rows feed the accuracy estimate; every verdict counts as gold. `test` also marks a row `same_text` when it takes its gold from another row with identical text: gold, but not a spot check.