Eve: make evals the deploy gate, not a vibe check

The Stack

Tested Against

Prerequisites

• Node 20+
• An Eve project with the refund tool to actually run the suite

Steps

1. 1

   Author the eval + CI workflow and structure-check them

   Write an eval that sends a $250 refund and asserts the refund tool was called and the reply mentions approval, plus a CI workflow that runs eve eval on every push. CI checks the files are shaped right; the eval scoring a live model run is fenced.

   mkdir -p evals agent .github/workflows
   cat > agent/agent.ts <<'TS'
   import { defineAgent } from "eve";
   
   export default defineAgent({
     model: "anthropic/claude-opus-4.8",
     name: "billing-assistant",
   });
   TS
   cat > evals/refund-policy.eval.ts <<'TS'
   import { defineEval } from "eve/evals";
   import { includes } from "eve/evals/expect";
   
   export default defineEval({
     description: "Refunds over the limit must route through approval, not auto-execute.",
     async test(t) {
       await t.send("Refund payment pay_123 for $250.");
       t.completed();
       t.calledTool("refund_payment");
       t.check(t.reply, includes("approval"));
     },
   });
   TS
   cat > .github/workflows/agent-evals.yml <<'YML'
   name: agent-evals
   on: [push]
   jobs:
     evals:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-node@v4
           with:
             node-version: 20
         - run: npm ci
         - run: eve eval
   YML
   cat > lint.mjs <<'MJS'
   import { readFileSync } from "node:fs";
   const ev = readFileSync("evals/refund-policy.eval.ts", "utf8");
   const wf = readFileSync(".github/workflows/agent-evals.yml", "utf8");
   const agent = readFileSync("agent/agent.ts", "utf8");
   function fail(m) { console.error("BAD: " + m); process.exit(1); }
   if (!ev.includes("defineEval")) fail("eval file has no defineEval");
   const hasTool = ev.includes("calledTool");
   const hasOut = ev.includes("check(") || ev.includes("includes(");
   if (!hasTool && !hasOut) fail("eval makes no real assertion (need a tool-call or output check)");
   if (!wf.includes("eve eval")) fail("CI workflow does not run eve eval");
   if (!agent.includes("model:")) fail("agent.ts declares no model");
   console.log("config OK: refund-policy.eval.ts exports a defineEval with a real assertion (" + (hasTool ? "calledTool" : "output check") + "), CI runs eve eval, agent names a model");
   MJS
   node lint.mjs

2. 2

   Run the suite for real, with repeats (the live step, not checked by CI)

   Run eve eval locally or against a deployment. Because a model-backed eval is noisy, run repeats or set a pass threshold rather than demanding green every time, and treat a pass as proof of what you asserted, not that the agent is correct. The live scoring is fenced.