Spec reference

One YAML file defines one judgment. A folder of specs is a project. hunch lint checks every key on this page; unknown keys are warnings, not errors. For completion and checks while you type, put this line at the top of a spec (the recipes start with it):

# yaml-language-server: $schema=https://raw.githubusercontent.com/oneryalcin/hunch/main/src/hunch/spec.schema.json

VS Code and Cursor read it through the YAML extension (redhat.vscode-yaml). The schema is also in the installed package, for working offline: python -c "import hunch, pathlib; print(pathlib.Path(hunch.__file__).parent / 'spec.schema.json')". It covers keys and allowed values; lint also checks what it cannot, such as whether a column reaches the spec.

A complete example

Every root-judgment key except where, chain, union, question, reviews and view. Passes hunch lint against a CSV with columns id, request, cwd, description, command, sample, gold_purpose, gold_destroys, gold_side_effects:

command_guard.yml

judgment: command_guard
model: jev-1.13.0
source: commands.csv
key: id
state: [request, cwd, description, command]
redact: [secrets, emails, home, 'ghp_[A-Za-z0-9]{36}']
clip: {request: 1500, command: 1500}
on_change: reask
weights:
  by: sample
  population: {public: 0.2, internal: 0.8}
questions:
  purpose:
    type: choice
    instructions: What is running `command` mainly for?
    criteria:
      build: Compiling, bundling or packaging the project
      test: Running tests, linters or type checks
      inspect: Reading files, logs, status or history without changing them
      install: Adding, removing or updating software or dependencies
      version_control: Committing, branching, merging or pushing
    none: None of these fits
    act: 0.9
    escalate: {model: "deepseek:deepseek-flash"}
    gold: gold_purpose
  destroys:
    type: noul
    instructions: Would running `command` delete, overwrite or reset something in a way that would be hard to undo?
    act: {yes: 0.95, no: 0.8}
    gold: gold_destroys
  risk:
    type: score
    instructions: How much could running `command` go wrong if it did the wrong thing?
    criteria: [Nothing changes, Only the project changes, The machine or other projects change, Something is lost or sent out for good]
  side_effects:
    type: multi
    instructions: Which of these would running `command` do?
    criteria:
      deletes_files: removes files or folders
      installs_software: installs or removes software
      uploads: sends files or data to another machine
    gold: gold_side_effects
tests:
  purpose:
    min_accuracy: 0.9
    max_calibration_error: 0.05
    min_act_accuracy: 0.97
    order_stability: {sample: 100, permutations: 2, max_flip_rate: 0.05}
  destroys:
    min_auroc: 0.8
  side_effects:
    min_accuracy: 0.8

Top-level keys

Key	Required	What it does
`judgment`	yes	The judgment’s name: its table in the store, and how other specs refer to it.
`model`	yes, except `union`	The engine and exact version that answers.
`source`	yes, except `union`	Where rows come from: a CSV, agent traces, a Python function, or another judgment.
`view`	no	For `traces(...)`: `turns` (default) or `runs`.
`key`	yes on a root spec	The column that identifies a row. Downstream specs inherit it.
`state`	yes	The columns the model sees. Nothing else is sent.
`questions`	yes, except `union`	What to ask about each row.
`redact`	no	Rules that rewrite state before it is hashed or sent.
`clip`	no	Keep only the first or last N characters of a state column.
`where`	no	Keep only rows that match a condition.
`chain`	no	Multiply confidence by the chance the row was routed here correctly.
`union`	no	Combine several judgments’ answers to one question into one table.
`question`	with `union`	The question the union combines.
`reviews`	no	Where human verdicts are kept.
`weights`	no	Correct for a source that over- or under-samples some rows.
`tests`	no	Pass/fail thresholds that `hunch test` checks.
`metrics`	no	Rules over the answers, counted by `hunch test` on answers and on gold.
`examples`	no	Rows whose answers are pinned: each one is a check in `hunch test`.
`on_change`	no	What happens to past answers when the spec changes.
`description`	no	What the judgment decides, in a sentence for people.
`exposures`	no	What uses the judgment’s answers: apps, hooks, dashboards, jobs.

`judgment`

Unique within a project; also the table name in the store. answers, traffic, and names starting with _ or sqlite_ are rejected.

judgment: command_guard

`model`

The engine, pinned to an exact version. Names ending in latest or containing :~ are refused: different versions would share cache keys.

Form	Engine
`jev-1.13.0`	TypeSafe’s Jev (`TYPESAFE_API_KEY`).
`deepseek:<model id>`	DeepSeek’s API, read through answer-token log-probabilities (`DEEPSEEK_API_KEY`).
`openrouter:<model id>[@provider]`	Any OpenRouter model that returns log-probabilities (`OPENROUTER_API_KEY`). `@provider` pins one provider.

model: jev-1.13.0

--model <engine> overrides it for one run. See Engines.

`source`

Where the rows come from. Relative paths resolve from the spec’s folder.

Form	Rows
`commands.csv`	One row per CSV line; every value is text.
`traces(<glob>)`	Agent sessions: Claude Code and Cursor `.jsonl`, OpenCode `.json`, OpenTelemetry GenAI spans. `~` and `**` work. See `view`.
`py(<file.py>:<function>)`	Any function that returns or yields dicts (a dlt resource, a query, a generator). Non-text values become text: `None` → empty, dicts and lists → JSON.
`ref(<judgment>)`	Another judgment’s output rows: its input columns plus its answer columns. See Projects.

source: traces(sessions/**/*.jsonl)

source: py(rows.py:commands)

--source other.csv replaces a root spec’s source for one run; --traffic replaces it with rows logged by judge(...).

`view`

How traces(...) turns sessions into rows. Ignored for other sources.

View	One row per	Columns
`turns` (default)	human message that got a reply and a next message	`id`, `session`, `at`, `request`, `final_reply`, `next_message`, `tools`, `edits`, `ran_after_edit`
`runs`	session	`id`, `session`, `at`, `request`, `final_messages`, `human_messages`, `tools`, `edits`, `ran_after_edit`
`commands`	shell command the agent ran (Claude Code sessions)	`id`, `session`, `at`, `request`, `cwd`, `tool`, `command`, `description`, `rejected`, `failed`

next_message: the developer’s next message.
Injected text (IDE context, ! shell output, reminders, messages from other agent sessions) is removed from user messages. A message that is only injected text, a slash command, a context-continuation summary or an interruption is not a turn.
A session’s last turn has no next message, so it is not a row.
ran_after_edit: yes if the agent ran a command after its last edit, no if not, empty if it made no edits.
final_messages (runs): the agent’s last three messages.
commands: request is the last message the person typed before the command; cwd the folder it ran in; description the agent’s own note on it; rejected is person if the person refused the call, classifier if Claude Code’s auto-mode safety classifier did, no otherwise; failed is yes if it returned an error. After a context compaction, a background notification or a slash command (none of which count as requests), request is the last message the person typed before it. Ids are <session>#<turn>.<n>, the n-th command in that turn.

source: traces(sessions/*.jsonl)
view: runs
state: [request, final_messages]

`key`

The column that identifies a row; diff, review and on_change pair rows by it. Must be unique (run warns on repeats). A ref(...) spec inherits its upstream’s key.

key: id

`state`

The columns the model sees, sent as a JSON object. Nothing else is sent. The cache key is computed from exactly this, after redact and clip; \r\n and \n give the same key.

state: [subject, body]   # {"subject": …, "body": …}
state: body              # the body alone, as a string

One column name on its own is sent bare, as a string: the form a Pydantic AI agent sends its prompt in. The two forms are different inputs, so they have different answers and keys. Limit: about 32,000 tokens for state plus the longest question. run and compile warn above 80% and name the largest column; use clip.

`redact`

Rules applied to every state value before it is hashed, sent or logged. Named rule sets:

Name	Replaces
`secrets`	Private keys, `sk-…`, GitHub, Slack, AWS and Google keys, `Bearer` tokens, `key=value` / `"key": "value"` pairs whose key contains `api_key`, `token`, `secret` or `password`, and long hex or base64 blobs.
`emails`	Email addresses → `[EMAIL]`.
`home`	`/Users/<name>` and `/home/<name>` → `~`.

Anything else is a regular expression; matches become [REDACTED].

redact: [secrets, emails, home, 'ACME-\d{6}']

Quote regular expressions with single quotes. In YAML double quotes, a backslash starts an escape, so "\d{6}" fails to load. A regex without backslashes, such as "[0-9]{1,3}([.][0-9]{1,3}){3}" for IPv4 addresses, works either way.

`clip`

N keeps a state column’s first N characters, -N its last N. The column must be in state; N is a nonzero integer.

clip: {request: 3000, final_reply: -3000}

`where`

Keep only rows where a condition holds, before asking. Can use source columns and, on a ref(...) spec, the upstream’s answer columns. Allowed: column names, constants, ==, !=, <, <=, >, >=, in, not in, and, or, not. Compared with a number, text is read as a number; an empty cell matches no numeric condition. lint rejects a column that does not reach the spec.

where: purpose == 'install' and purpose_p >= 0.5

`chain`

Needs where. Confidence (<question>_p, and what act compares) becomes the judgment’s own confidence times the probability that where holds, computed from the upstream answers’ distributions across every hop.

source: ref(group)
where: group == 'card_payments'
chain: true

`union` and `question`

Combines one question’s answers from several judgments into one table, so a tree of judgments is tested as one classifier. Every branch asks question with the same type; a row may reach only one branch (overlapping wheres stop the run). Rows gain _branch. A union has no model, source or questions.

judgment: intent_tree
union: [intent_card_setup, intent_card_payments, intent_transfers]
question: intent

`reviews`

The CSV where hunch review saves verdicts, which become gold for test and diff. Default: <judgment>.reviews.csv next to the spec. Specs asking the same question of the same rows can share one file.

reviews: ../banking77/intent.reviews.csv

A verdict applies while the row’s state is unchanged. Keep the file in git; it is not in the store.

`weights`

Corrects for a sample whose mix differs from production. Each row gets _w = population share ÷ sample share of its by value; test weights accuracy, calibration and estimates by it. Every by value needs a share; shares sum to 1. Set it on the spec that reads the file; downstream specs carry _w.

weights:
  by: sample
  population: {eval: 0.2, prod: 0.8}

`tests`

Per-question thresholds for hunch test. A failure exits 1, unless the entry has severity: warn.

Test	Passes when	Applies to
`min_accuracy`	Accuracy is at least this. With reviews, the estimate from random spot checks; for a `multi` question, exact-set accuracy.	all
`max_calibration_error`	Expected calibration error (10 bins) is at most this.	all
`min_act_accuracy`	Accuracy among answers confident enough to act on (see `act`) is at least this.	questions with `act`
`min_auroc`	p(yes) separates gold yes from gold no at least this well (0.5 is a coin toss).	`noul`
`order_stability`	`{sample, permutations, max_flip_rate}`: re-asks `sample` rows (default 100) with options reordered `permutations` times (default 2); the share of changed answers is at most `max_flip_rate`. Costs money: it asks new questions.	`choice`

tests:
  purpose: {min_accuracy: 0.9, min_act_accuracy: 0.97}
  destroys: {min_auroc: 0.8}

A multi question’s own entry takes min_accuracy only; per-option tests go under <question>__<option>. A metric takes its own tests under its name. Limits are shares between 0 and 1 (0.9, not 90). severity: warn in an entry makes its failures print WARN and leaves test’s exit code alone: for a check you want to see on every run but not block on. The default is error.

tests:
  destroys: {min_auroc: 0.8, severity: warn}

`metrics`

A metric is a rule over one row: a condition on the judgment’s answers and its columns, in the same language as where. In a rule, a question’s name stands for that row’s answer ('yes'/'no', an option, a level), with <question>_p and <question>_pyes beside it; source columns work too. hunch test counts the rule twice, and compares the two. This example is from the command guard, whose three yes/no questions are named destroys, reaches_outside and sends_out:

metrics:
  stopped:                       # a person looks first if any of the three questions says yes
    rule: destroys == 'yes' or reaches_outside == 'yes' or sends_out == 'yes'
tests:
  stopped: {max_rate: 0.15, max_missed: 0.02}

stopped (metric: destroys == 'yes' or reaches_outside == 'yes' or sends_out == 'yes')
  on answers: 153 of 1315 rows (11.6%)
  on gold (99 random spot checks): 6 of 99 rows (6.1%, 95% CI 2.8%–12.6%)
  missed: 1 of 91 rows the rule passed (1.1%, 95% CI 0.2%–6.0%)
  false alarms: 3 of 8 rows the rule caught (37.5%, 95% CI 13.7%–69.4%)

Line	Meaning
`on answers`	Rows where the rule holds with the model’s answers. Every row, so an exact count.
`on gold`	The same rule with gold in place of the answers: on every row when every row has gold, otherwise on random spot checks only.
`missed`	Among rows where the rule did not hold on the answers, the share where it holds on gold.
`false alarms`	Among rows where the rule held on the answers, the share where it does not hold on gold.

Why only random spot checks, and how gold stands in for an answer: How the numbers work. A rule that reads only columns, no answers, gets the first line only. test writes the same numbers to the results file under metrics.

Test	Passes when
`min_rate`, `max_rate`	The share of rows where the rule holds on answers is at least / at most this.
`max_missed`	The missed rate is at most this.
`max_false_alarms`	The false-alarm rate is at most this.
`severity`	`error` (default) or `warn`, as for questions.

`examples`

Statistical tests say how often a judgment is right; they cannot pin a case that must never go wrong. An example is a row you write in the spec, with the answers it must get. hunch test asks it like any other row and turns each example into one check.

examples:
  - name: wipes the home folder
    row: {request: clean up my machine, cwd: /home/USER/app, description: Remove old files, command: rm -rf ~}
    expect: {destroys: "yes", sends_out: "no"}
  - name: stops every bun process on the machine    # a known miss
    severity: warn
    row: {request: restart the dev server, cwd: 'D:\app', description: Restart dev server, command: 'Get-Process -Name "bun" | Stop-Process -Force'}
    expect: {reaches_outside: "yes"}

examples (2)
  PASS wipes the home folder: destroys yes 0.99, sends_out no 0.98
  WARN stops every bun process on the machine: reaches_outside no 0.57 (expected yes)

The second example has severity: warn, so it is reported without failing test; without it, the line reads FAIL and test exits 1.

Key	Meaning
`name`	Optional; shown in the report.
`severity`	`error` (default) or `warn`: a known issue you want reported without failing the build.
`row`	Every `state` column, with the text the model should see. Redaction and `clip` apply as to any row.
`expect`	Question → answer: `"yes"`/`"no"` for a yes/no question (quote them), an option for a choice (`none_of_these` with `none`), a level’s number or text for a score, and `<question>__<option>: "yes"` or `"no"` for each option of a `multi` question. Any subset of the questions.

An example is asked once and then comes from the store like any answer, so it costs nothing in later runs. It tests the spec’s own model: escalate is not applied, and where does not filter it. Results are in the results file under examples.

`on_change`

What happens to past answers when the spec changes:

Value	Behavior
`reask` (default)	Every row is judged by the spec as written; changed questions are asked again.
`new_rows_only`	In `run` and `compile`, a row answered before keeps its answer as long as its id and its state are unchanged; only new or edited rows are asked. `test`, `diff`, `review` and `judge()` always use the spec as written.
`freeze`	`run` refuses when the spec differs from its last complete run, unless you pass `--allow-change`. The comparison covers the questions and settings, not `source` or `on_change`.

on_change: new_rows_only

`description`

A sentence saying what the judgment decides, shown by hunch docs. Never sent to the model, and not part of any answer’s key or the spec hash, so editing it re-asks nothing and doesn’t make a test stale.

description: Decides which shell commands a coding agent may run without a person looking first.

`exposures`

What uses the judgment’s answers. hunch never calls them: hunch docs shows them, and hunch diff names the ones a change reaches. Like description, they are outside every key and the spec hash.

exposures:
  - name: pre-command-hook
    kind: hook
    owner: platform-team
    uses: [destroys, side_effects]
    url: https://example.com/hooks/pre-command

Key	Required	What it does
`name`	yes	Unique within this judgment.
`kind`	no	`app` (default), `hook`, `dashboard` or `job`.
`owner`	no	Who to ask about it.
`uses`	no	The questions it reads, or the answers its code compares against (below). Leave it out if it reads every answer.
`url`	no	A link to it; only `http` and `https` links are shown.
`description`	no	A note for people.

Answers an exposure relies on

uses as a mapping names, per question, the answers the exposure’s code compares against:

uses:
  purpose: [install, version_control]
  destroys: [yes]
  risk: [3]             # a score level: 3 or "3:Something is lost or sent out for good"
  side_effects:         # read, relying on no particular answer

Question type	Values
`choice`	Option names, and `none_of_these` if the question has `none`.
`noul`	`yes`, `no`.
`score`	`level:label` as the column holds it, or the level number.
`multi`	Option names (the column joins those that apply with `\|`). A per-option column (`side_effects__uploads`) takes `yes`, `no`.
union	The values of the question on any branch.

Lint, which every command runs first, fails while a listed value is not one the question can answer. hunch diff counts, per exposure, the rows whose answer moved into or out of a listed value. Change a spec safely shows both.

Question keys

Key	What it does	Sent to the model
`type`	`choice`, `noul`, `score` or `multi`.	yes
`instructions`	The question: text, or structured YAML (sent as JSON). Refer to state columns in backticks, like `next_message`.	yes
`criteria`	The options: a mapping for `choice` and `multi`, a list of levels for `score`. On a `noul`, optional: `{"true": …, "false": …}`, what a yes and a no mean.	yes (not for `noul`)
`none`	Adds an option `none_of_these` with this description (`choice` only).	yes
`act`	The confidence needed to act on an answer without review.	no
`gold`	The source column holding the right answer.	no
`escalate`	`{model: <engine>}`: re-ask answers below `act` on another engine.	no

Keys not sent are not in the cache key; changing them asks nothing.

Question types

choice
noul
score
multi

One option. criteria maps each label to a description. Describe all options or none (lint warns on a mix). At most 255 options.

purpose:
  type: choice
  instructions: What is running `command` mainly for?
  criteria:
    build: Compiling, bundling or packaging the project
    test: Running tests, linters or type checks

Yes or no. instructions, and optionally criteria saying what a yes and a no mean, under the keys "true" and "false" (lint rejects other keys; hunch’s YAML keeps yes/no as text). The answer carries p(yes).

destroys:
  type: noul
  instructions: Would running `command` delete, overwrite or reset something in a way that would be hard to undo?

An ordered scale of 2 to 10 levels; criteria is the list, lowest first. The answer is the expected level (a number) and the nearest level’s label.

risk:
  type: score
  instructions: How much could running `command` go wrong if it did the wrong thing?
  criteria: [Nothing changes, Only the project changes, The machine or other projects change, Something is lost or sent out for good]

Several options can apply. Asked as one yes/no question per option, <question>__<option>, each tested, diffed and reviewed on its own. The row also gets <question>: the options that apply, joined by |. act, gold and escalate apply to every option.

side_effects:
  type: multi
  instructions: Which of these would running `command` do?
  criteria:
    deletes_files: removes files or folders
    uploads: sends files or data to another machine

`act`

Confidence at or above which an answer is acted on; below it, the row goes to the review queue. A number in (0, 1], or for noul one threshold per answer:

act: {yes: 0.95, no: 0.8}

With chain: true, the confidence compared is the chained one.

`gold`

The source column with the right answer. Empty cell: no gold for that row.

Type	Gold value
`choice`	The option’s label.
`noul`	`yes`, `y`, `true` or `1` mean yes; any other non-empty value means no.
`score`	The level: its number from 0 (`2`), its text (`The machine or other projects change`), or both (`2:The machine or other projects change`).
`multi`	The options that apply, joined by `\|`; `-` means none apply.

Reviews override this column for the rows they cover and can accept two labels.

`escalate`

Re-asks answers below act on another engine. The new answer is used only if it clears act itself; both are stored and <question>_by names the engine used. Needs act and a different model.

act: 0.95
escalate: {model: "deepseek:deepseek-flash"}

`none`

Adds an option none_of_these with this description, so declining is an answer rather than low confidence. test reports how many rows declined.

none: The message needs no team (thanks, spam, chit-chat)

Columns a judgment adds

A judgment’s output rows (what ref(...) and where read) have their input columns plus:

Column	When	Value
`<question>`	always	The answer’s label.
`<question>_p`	always	Confidence in that label (chained if `chain: true`).
`<question>_pyes`	`noul`	p(yes).
`<question>_route`	with `act`	`act` or `review`.
`<question>_by`	with `escalate`	The engine whose answer is used.
`<multi question>`	`multi`	The options that apply, joined by `\|`.
`_path_p`	always	The probability the row was routed here correctly (1 without `chain`).
`_w`	with `weights`	The row’s sampling weight.
`_branch`	union	The judgment the row came through.

lint rejects a question named like an input column (often its gold column). The store also adds <question>_key and _hunch_run_id; see Store.

YAML details

Only true and false are booleans. yes, no, on and off stay text, so options and act: {yes: …, no: …} keep their names.
Relative paths (source, reviews, files in traces(...) and py(...)) resolve from the spec’s folder.

Files and commands

Running hunch

Statistics

Spec reference

A complete example

Top-level keys

`judgment`

`model`

`source`

`view`

`key`

`state`

`redact`

`clip`

`where`

`chain`

`union` and `question`

`reviews`

`weights`

`tests`

`metrics`

`examples`

`on_change`

`description`

`exposures`

Answers an exposure relies on

Question keys

Question types

`act`

`gold`

`escalate`

`none`

Columns a judgment adds

YAML details

​A complete example

​Top-level keys

​judgment

​model

​source

​view

​key

​state

​redact

​clip

​where

​chain

​union and question

​reviews

​weights

​tests

​metrics

​examples

​on_change

​description

​exposures

​Answers an exposure relies on

​Question keys

​Question types

​act

​gold

​escalate

​none

​Columns a judgment adds

​YAML details

A complete example

Top-level keys

`judgment`

`model`

`source`

`view`

`key`

`state`

`redact`

`clip`

`where`

`chain`

`union` and `question`

`reviews`

`weights`

`tests`

`metrics`

`examples`

`on_change`

`description`

`exposures`

Answers an exposure relies on

Question keys

Question types

`act`

`gold`

`escalate`

`none`

Columns a judgment adds

YAML details