Sakana Fugu: A/B it on your own task before you migrate

The Stack

Steps

1. 1

   Write the client config and validate it

   Point your client at https://api.sakana.ai/v1 with model fugu or fugu-ultra. For regulated data use the fugu variant and list providers to opt out of the pool (Fugu Ultra's pool is fixed, so opt-out is rejected there). CI checks the config shape; the run needs your key and is fenced.

   cat > client.json <<'JSON'
   {
     "base_url": "https://api.sakana.ai/v1",
     "model": "fugu",
     "exclude_providers": ["provider-x", "provider-y"]
   }
   JSON
   node -e '
   const fs = require("fs");
   const c = JSON.parse(fs.readFileSync("client.json", "utf8"));
   function bad(m) { console.error("BAD: " + m); process.exit(1); }
   if (!c.base_url || !String(c.base_url).includes("api.sakana.ai")) bad("base_url does not point at the Fugu endpoint");
   const m = c.model || "";
   const isFugu = m === "fugu";
   const isUltra = m.indexOf("fugu-ultra") === 0;
   if (!isFugu && !isUltra) bad("model must be fugu or fugu-ultra");
   if (Object.prototype.hasOwnProperty.call(c, "exclude_providers")) {
     if (isUltra) bad("exclude_providers is only valid for the fugu variant (Fugu Ultra pool is fixed)");
     if (!Array.isArray(c.exclude_providers) || !c.exclude_providers.every((x) => typeof x === "string")) bad("exclude_providers must be a list of strings");
   }
   const optout = isFugu && Array.isArray(c.exclude_providers) ? c.exclude_providers.length : 0;
   console.log("config OK: client points at the Fugu endpoint, model " + m + ", opt-out list of " + optout + " provider(s)");
   '

2. 2

   Run the A/B (the model step, not checked by CI)

   Set your Sakana key, run one hard task you have a known-good answer for on Fugu Ultra and on a single strong model, and compare quality, latency, and per-call cost. Migrate only if it wins on your work. The runs are fenced.