monitoring-liveness-watch

Monitoring liveness watch — who watches the watchers

Hourly: finds exporters that are up but blind, expected metrics and log streams that stopped existing, dashboard panels showing nothing, alert rules that cannot fire or deliver, and watcher workflows that went silent. One issue per gap, symptoms folded into their cause, uncertain ones to a person.


Source	`templates/monitoring-liveness-watch/workflow.yaml`
Category	`debugging-ops`
Trigger	manual (`wfx run`) · schedule `7 * * * *`
Steps	6
Human approval before	none
Providers named	`claude-cli`

Use it

wfx templates monitoring-liveness-watch
wfx new monitoring-liveness-watch --as my-monitoring-liveness-watch
wfx dryrun my-monitoring-liveness-watch

What it does

Watches the monitoring itself: exporters that are up but cannot read their source, expected metrics and log streams that stopped existing, dashboards whose panels show nothing, alert rules that cannot evaluate or deliver, and watcher workflows whose heartbeat went stale. Each gap is explained against the repository, judged by a calibrated classifier, and filed as one issue.

What you fill in

env: PRODUCT, ISSUE_REPO, ISSUE_LABEL — use the SAME label as your prod-watch so the two dedupe against each other
collect step env: OBS_SSH / PROM_URL / LOKI_URL, and which sources to check: EXPECTED_METRICS, EXPORTER_QUERIES (glob of exporter queries.yaml in your repo), DASHBOARDS (glob of Grafana JSON in your repo), HEARTBEATS (watcher workflow=max minutes)
checks/: the shipped ones are generic (exporter self-errors, rule evaluation, alert delivery, config reload); add a *.logql with raise: empty per log stream you rely on (see logs_flowing.logql.example)
expected-metrics.txt: your custom metrics, one selector per line
triage skills: add your product’s production-map skill beside prod-signal-triage
the project env store needs TYPESAFE_API_KEY (or OPENROUTER_API_KEY) for the classifier

Inputs

Input	Type	Default	Required	Meaning
`dry_run`	boolean	`false`

Steps

#	Step	Kind	Skills	Gates
1	`collect` Check that every signal still exists	shell (`run:`)		`exitCode` = `2` → fail
2	`heartbeat` Write the heartbeat	shell (`run:`)
3	`triage` Explain each gap against the repository	agent on `claude-cli`	`prod-signal-triage`
4	`verdict` Judge each gap (JEV)	shell + classifier (`wfx judge`)		`exitCode` = `2` → fail
5	`file` File one issue per gap	shell (`run:`)		`exitCode` = `1` → fail
6	`escalate` Ask a person about the uncertain ones	agent on `claude-cli`	`approval-desk`

Human gates

escalate may call ask_human; answer with wfx answer <run-id> -m "...".