plan-execute-judge

A generic plan→execute→judge pipeline for the Claude Agent SDK or Codex SDK, with an optional deep-research phase in front. Fresh agent runs with real control flow — the research brief, the plan, and a typed verdict are the only data crossing phase boundaries. Never a transcript.

TypeScript Node ≥ 22.9 MIT Claude Agent SDK Codex SDK

The pipeline

task → research
optional → plan → execute → judge
typed verdict

↳ a failing verdict feeds its gaps into the next execute round — up to maxRounds, then exit 1

Research and planning can fan out across multiple independent agents; when several plans are produced, a refinements phase merges them into one final PLAN.md.

Execute implements the plan under acceptEdits and leaves changes uncommitted. Judge starts cold — only the plan, the task, and what's on disk — and diffs against a recorded baseline.

Design decisions

The judge never sees execute's transcript

Each phase is its own query with no resume, so the judge starts cold with only the plan, the task, and whatever's on disk — not execute's narrative of what it did. That's the part that keeps it from rubber-stamping.

The verdict is structured, not prose

Judge output uses a JSON schema — { pass, summary, gaps[] } — so verdict.pass is a real boolean the orchestrator branches on. A narrow schema avoids style commentary and scope creep on every round.

Bash is check-only, edits are gated

Plan and judge run under a deny-by-default permission mode with a Bash-vetting hook: inspection and test commands are allowed; git-state mutation, dependency installs, network fetches, and output redirection are denied. Defense-in-depth, not a sandbox.

The judge checks the task, not just the plan

The original task is in the judge's prompt: if the implementation satisfies the plan but the plan missed part of the task, that's a failing gap. Otherwise a bad plan converges confidently on the wrong thing.

Configuration

Everything is driven by environment variables — backend, model, effort, round count, and multi-agent fan-out.

env varmeaningdefault

PEJ_BACKENDAgent runtime — claude or codexclaude

PEJ_MODELModel id for every phaseper-backend

PEJ_MAX_ROUNDSExecute→judge cycles before giving up3

PEJ_PLAN_AGENTSIndependent planning agents before refinements1

PEJ_RESEARCH_SOURCESComma-separated URLs, repos, PDFs to researchunset

Run it

Run from inside the git repo you want it to work on. Exit 0 means the judge passed; changes are left uncommitted for you to review.

$ npm install && npm test

$ npm start -- "add rate limiting to the /upload endpoint, 10 req/min per IP"

← all projects github.com/glslang/plan-execute-judge ↗