Pick a model with evidence: a GitHub Models bake-off that fits the free cap

The Stack

Steps

1. 1

   Write the bake-off plan and prove it fits the free cap

   List the prompts that actually matter, the models to compare, and how you will pick the winner. CI checks the plan is a real comparison and that prompts times models stays under the free daily cap, so you do not get rate-limited halfway through. Running the prompts needs your GitHub login and is fenced.

   cat > bakeoff.json <<'JSON'
   {
     "prompts": ["summarize ticket", "extract fields", "classify intent", "draft reply", "rate sentiment"],
     "models": ["gpt-5.5", "deepseek-r1", "llama-4-maverick"],
     "select_by": "win_rate",
     "free_daily_cap": 50
   }
   JSON
   node -e '
   const fs = require("fs");
   const c = JSON.parse(fs.readFileSync("bakeoff.json", "utf8"));
   function bad(m) { console.error("BAD: " + m); process.exit(1); }
   const P = (c.prompts || []).length;
   const M = (c.models || []).length;
   if (P < 2) bad("need at least 2 prompts to compare meaningfully");
   if (M < 2) bad("need at least 2 models for a bake-off");
   if (!c.select_by) bad("no selection criterion (how do you pick the winner?)");
   const cap = c.free_daily_cap || 0;
   const calls = P * M;
   if (cap && calls > cap) bad("bake-off needs " + calls + " calls but the free cap is " + cap + "/day");
   console.log("config OK: bake-off of " + P + " prompt(s) x " + M + " model(s) = " + calls + " call(s), within the " + cap + "/day free cap; pick by " + c.select_by);
   '

2. 2

   Run the prompts and keep the winner (the model step, not checked by CI)

   Point the same prompts at each model through GitHub Models, score them by your chosen metric, and keep the winner. Then move that workload to whichever provider hosts your pick. The runs and scoring are fenced.