Advisor pattern: cap how often the expensive model gets called, and catch drift

The Stack

Steps

1. 1

   Declare the call budget and drift-check, and compute the real effective cost

   Set a hard cap on advisor calls for the task, require a drift-check every N executor steps before it can continue without a fresh check-in, and record the benchmark's assumed advisor rate next to your actual one. CI checks the budget and drift-check are present and computes effective cost from the real call count, not the assumed rate. Running the executor and advisor is fenced.

   cat > advisor-task.json <<'JSON'
   {
     "executor": { "model": "sonnet-5", "usd_per_mtok_out": 15 },
     "advisor": { "model": "fable-5", "usd_per_mtok_out": 50 },
     "max_advisor_calls": 3,
     "drift_check_every_n_steps": 5,
     "task": { "steps": 20, "advisor_calls_actual": 2 },
     "benchmark_assumed_advisor_rate": "about once per task"
   }
   JSON
   node -e '
   const fs = require("fs");
   const c = JSON.parse(fs.readFileSync("advisor-task.json", "utf8"));
   function bad(m) { console.error("BAD: " + m); process.exit(1); }
   if (!c.max_advisor_calls || c.max_advisor_calls < 1) bad("set a hard max_advisor_calls, an uncapped advisor erodes the discount");
   if (!c.drift_check_every_n_steps || c.drift_check_every_n_steps < 1) bad("set drift_check_every_n_steps so the executor cannot run unchecked forever");
   const t = c.task || {};
   const actual = t.advisor_calls_actual;
   if (actual == null) bad("record advisor_calls_actual so you can compare it to the cap and the benchmark assumption");
   if (actual > c.max_advisor_calls) bad("advisor_calls_actual (" + actual + ") exceeds max_advisor_calls (" + c.max_advisor_calls + "), the budget was not enforced");
   const steps = t.steps || 0;
   const checksNeeded = Math.floor(steps / c.drift_check_every_n_steps);
   if (checksNeeded > 0 && actual < 1) bad("task has enough steps to require a drift-check but no advisor calls were made");
   const exec = c.executor.usd_per_mtok_out, adv = c.advisor.usd_per_mtok_out;
   const mixShare = actual / (steps || 1);
   const effRate = ((1 - mixShare) * exec + mixShare * adv) / adv;
   console.log("advisor OK: cap " + c.max_advisor_calls + ", actual " + actual + " call(s) over " + steps + " steps, drift-checked every " + c.drift_check_every_n_steps + "; effective cost ~" + Math.round(effRate * 100) + "% of pure-advisor rate (benchmark assumed " + c.benchmark_assumed_advisor_rate + ")");
   '

2. 2

   Run the task and watch the real call count (the model step, not checked by CI)

   Run the executor with the advisor wired in, honoring the cap and the drift-check. Compare advisor_calls_actual to the benchmark's assumed rate after the run: if your task pulled the advisor in more often, your real discount is smaller than 63%. If the job turns out to be easy, skip the pattern next time and run the cheap model alone. The run and the judgement are fenced.