Self-Healing Test Loop: an Agent that Fixes Until Green

The Stack

Tested Against

Prerequisites

• Claude Code (claude -p) or Codex (codex exec)
• A real test command as the verifier (npm test, pytest, etc.)
• An API key in CI: ANTHROPIC_API_KEY for Claude Code, CODEX_API_KEY for Codex

Steps

1. 1

   The shape: agent, verifier, bounded stop

   The loop runs your tests; if they fail it pipes the failure to the agent to fix the code, and repeats until green or MAX attempts. Claude Code: pipe the output to claude -p with --allowedTools Read,Edit,Bash and --permission-mode acceptEdits. Codex: pipe to codex exec --sandbox workspace-write (the current flag; --full-auto is deprecated). The verifier is your real test command, the most trustworthy judge there is.

2. 2

   What CI checks: the loop stops on pass and is bounded

   CI models the loop's control flow with a stub agent (standing in for the fenced claude -p / codex exec) and asserts the two guardrails the newsletter insists on: the loop stops the instant the verifier passes, and a never-fixed problem stops after MAX instead of spinning forever. No model, no key.

   cat > checkloop.mjs <<'EOF'
   // Model the self-healing loop's control flow and assert its two guardrails:
   // it stops on the first passing verify, and it is bounded by MAX.
   const MAX = 5;
   function runLoop(fixer){
     let state = 'RED';
     const verify = () => state === 'GREEN';
     for (let i = 1; i <= MAX; i++){
       if (verify()) return { passed: true, attempt: i };
       state = fixer(i, state);
     }
     return { passed: false, attempt: MAX };
   }
   const fixNow = () => 'GREEN';   // stub agent that fixes on its first turn
   const neverFix = (i, s) => s;   // stub agent that never fixes
   let ok = true;
   function check(label, cond){ console.log(label + ': ' + (cond ? 'yes' : 'NO')); if(!cond) ok=false; }
   const a = runLoop(fixNow);
   check('loop stops on first passing verify (attempt 2)', a.passed === true && a.attempt === 2);
   const b = runLoop(neverFix);
   check('loop is bounded: stops after MAX with no infinite spin', b.passed === false && b.attempt === MAX);
   if(!ok){ console.log('self-healing loop check FAILED'); process.exit(1); }
   console.log('self-healing loop check OK');
   EOF
   node checkloop.mjs

3. 3

   Run it with your agent (the model step, not checked by CI)

   Point the loop at your repo and let claude -p or codex exec do the fixing each iteration. That call runs a model, so it is non-deterministic and fenced: CI proves the loop is bounded and stops on success, never the model's edits. Tell it to fix the code, not the tests.