test-gap-finder

Test-gap finder — exhaustive missing-test scenarios, triaged, proven PR

Enumerates everything a test could pin (every public function, branch, error path, boundary, README promise), finds which ones no test pins, has a calibrated classifier and a person decide which are worth a test, writes them in the repo’s own style, proves each one fails on a mutant, and opens one PR listing every scenario and its verdict.


Source	`templates/test-gap-finder/workflow.yaml`
Category	`development`
Trigger	manual (`wfx run`)
Steps	17
Human approval before	`write`, `publish`
Providers named	`haiku`

Use it

wfx templates test-gap-finder
wfx new test-gap-finder --as my-test-gap-finder
wfx dryrun my-test-gap-finder

What it does

Exhaustive test-gap finding. Code enumerates the whole testable surface with file:line and coverage; analysts account for every item, shard by shard; scenarios already pinned by a green test are marked covered; a calibrated classifier sorts the rest into makes_sense / does_not_make_sense / unsure, and a person decides the unsure ones. Approved scenarios become tests in the repo’s own style on a branch, proven by the full suite and a mutation check, reviewed by a different model, and published as one PR after a person approves.

What you fill in

allow_red_baseline: true only if the suite is already red and you accept freezing those failures
scope: one module (and spec_paths: its specs/ADRs) per run — exhaustive means every item, so keep the portion small
max_tests: how many tests one PR may add (default 20; the riskiest go first)
TYPESAFE_API_KEY or OPENROUTER_API_KEY in the env store for the calibrated verdicts

Inputs

Input	Type	Default
`repo_path`	string
`repo_url`	string
`base_branch`	string
`scope`	string
`max_tests`	integer	`20`
`spec_paths`	string
`allow_red_baseline`	boolean	`false`
`pr_prefix`	string	`"[test-gaps]"`
`dry_run`	boolean	`true`
`isolate`	boolean	`true`
`extra_context`	string

Steps

#	Step	Kind	Skills	Gates
1	`style` Learn the repo’s test style (repo-navigator)	agent	`repo-navigator` `test-author`
2	`baseline` Baseline suite, counts and coverage	shell (`run:`)		`exitCode` = `70` → fail `exitCode` = `78` → skip to `report`
3	`surface` Enumerate the testable surface	shell (`run:`)		`exitCode` = `70` → fail `exitCode` = `78` → skip to `report` `exitCode` = `11` → skip to `scenarios_1` `exitCode` = `12` → skip to `scenarios_2` `exitCode` = `13` → skip to `scenarios_3`
4	`scenarios_4` Scenario analyst — shard 4	agent	`test-author` `repo-navigator`
5	`scenarios_3` Scenario analyst — shard 3	agent	`test-author` `repo-navigator`
6	`scenarios_2` Scenario analyst — shard 2	agent	`test-author` `repo-navigator`
7	`scenarios_1` Scenario analyst — shard 1	agent	`test-author` `repo-navigator`
8	`merge` Merge and dedupe against the suite	shell (`run:`)		`exitCode` = `70` → fail `exitCode` = `78` → skip to `report`
9	`triage` Triage (JEV) — makes_sense / does_not_make_sense / unsure	shell (`run:`)		`exitCode` = `70` → fail `exitCode` = `20` → needs_input `exitCode` = `78` → skip to `report`
10	`write` Write the approved tests (test-author)	agent	`test-author`	human approval first
11	`validate` Prove the tests (suite + mutation + coverage)	shell (`run:`)		`exitCode` = `70` → fail `exitCode` = `20` → needs_input
12	`review` Adversarial review on a different model (change-reviewer)	agent on `haiku`	`change-reviewer`	`approved` = `false` → needs_input
13	`write_pr` Write the PR body from the evidence	shell (`run:`)		`exitCode` = `70` → fail
14	`claims` Re-check the body against the evidence	shell (`run:`)		`exitCode` = `20` → needs_input `exitCode` = `70` → fail
15	`dry_run_stop` Stop here on a dry run	shell (`run:`)		`exitCode` = `78` → skip to `report`
16	`publish` Push and open the PR (pr-publisher)	agent	`pr-publisher`	human approval first
17	`report` Report	shell (`run:`)

Human gates

write waits for wfx approve <run-id> (or wfx reject <run-id> -m "why") before it runs.
publish waits for wfx approve <run-id> (or wfx reject <run-id> -m "why") before it runs.
triage pauses for input when exitCode is 20: “{{ .Output.stdout }}”. Resume with wfx answer <run-id> -m "...".
validate pauses for input when exitCode is 20: “{{ .Output.stdout }}”. Resume with wfx answer <run-id> -m "...".
review pauses for input when approved is false: “The reviewer blocked the branch: {{ .Output.verdict }}”. Resume with wfx answer <run-id> -m "...".
claims pauses for input when exitCode is 20: “PR body does not match the evidence — {{ .Output.stdout }}”. Resume with wfx answer <run-id> -m "...".

Real result

It was run for real on python-streams, hand-drawn-diagrams and citenexus. Those runs changed the template itself: a scope input (one module plus its specs per run), and no PYTHONPATH injection, which had shadowed the installed project and broken 2 citenexus tests (commit 497a7f1).
On one run the analysts claimed 118 scenarios were “already covered”. Triage’s classifier read each named test and sent 9 back as gaps, because the test did not actually assert the scenario. The PR body now quotes the verified 109 and says why each claim was rejected (commit d2643e8).