workflow-eval

Workflow eval — did that run really do its job?

Point it at a finished run of any workflow. It builds a dossier of what the run was for and what it actually did, has four independent judges score it in parallel — a calibrated classifier and three different model families — and folds them into one scorecard with the disagreements called out.


Source	`templates/workflow-eval/workflow.yaml`
Category	`development`
Trigger	manual (`wfx run`)
Steps	9
Human approval before	none
Providers named	`gpt-oss`, `codex-cli`, `grok-cli`

Use it

wfx templates workflow-eval
wfx new workflow-eval --as my-workflow-eval
wfx dryrun my-workflow-eval

What it does

Evaluates a FINISHED run of any workflow. A command reads the run from the platform (its steps, outputs, errors, approvals, definition) and its activity log, checks that whatever it claims to have filed actually exists, and writes a dossier. Four judges read the same dossier independently and in parallel — JEV (a calibrated classifier: did it do the job, is it evidence-backed, would a senior engineer accept it, how bad is its worst problem) and three model judges on different providers (gpt-oss, codex-cli, grok-cli), each giving the same typed verdict. A judge that cannot run is probed first and recorded as UNAVAILABLE with the reason, never dropped. A final command builds the scorecard: agreement between judges, disagreements named, and an overall verdict that is PASS only when JEV says the run did its job AND at least two model judges pass it.

What you fill in

a rubric input when the workflow’s own description is not the bar you judge it by
the model judges’ providers, if gpt-oss, codex-cli or grok-cli are not the second opinions you trust
the overall rule in scorecard.py, if ‘JEV yes AND two model passes’ is not your bar

Inputs

Input	Type	Default	Required
`run_id`	string		yes
`rubric`	string
`api`	string	`"http://127.0.0.1:8090"`

Steps

#	Step	Kind	Gates
1	`dossier` Build the run’s dossier	shell (`run:`)	`ok` = `false` → fail
2	`probe_gptoss` Probe gpt-oss (OpenRouter)	shell (`run:`)
3	`probe_codex` Probe codex-cli	shell (`run:`)
4	`probe_grok` Probe grok-cli	shell (`run:`)
5	`jev` Judge — JEV (calibrated)	shell + classifier (`wfx judge`)
6	`judge_gptoss` Judge — gpt-oss	agent on `gpt-oss`
7	`judge_codex` Judge — codex-cli	agent on `codex-cli`
8	`judge_grok` Judge — grok-cli	agent on `grok-cli`
9	`scorecard` Build the scorecard	shell (`run:`)

Human gates

This template never stops for a person.