OrcaRouter for coding: judge by passing tests, not by vibes

The Stack

Steps

1. 1

   Author the tests_pass routing rule and lint it

   Write routing.yaml with a hard_code rule that fans out to a panel and resolves it with arbiter.strategy: tests_pass, so the patch that passes your tests wins. CI parses the DSL and asserts the strategy is tests_pass with a panel of at least two models.

   cat > routing.yaml <<'YAML'
   version: 1
   rules:
     - id: hard_code
       when: task_class == "code" && difficulty > 0.6
       use:
         parallel:
           - { model: "anthropic/claude-opus-4.8" }
           - { model: "openai/gpt-4o" }
         arbiter:
           strategy: tests_pass
           model: "anthropic/claude-opus-4.8"
         max_latency_ms: 120000
   default:
     delegate: cheapest
   YAML
   ruby -ryaml -e '
   c = YAML.safe_load(File.read("routing.yaml"))
   abort "BAD: version must be 1" unless c["version"] == 1
   rule = (c["rules"] || []).find { |r| r["id"] == "hard_code" }
   abort "BAD: no hard_code rule" unless rule
   use = rule["use"] || {}
   panel = use["parallel"]
   abort "BAD: hard_code has no panel" unless panel.is_a?(Array) && panel.length >= 2
   arb = use["arbiter"] || {}
   abort "BAD: arbiter strategy is not tests_pass" unless arb["strategy"] == "tests_pass"
   puts "config OK: hard_code fans out to a " + panel.length.to_s + "-model panel judged by tests_pass (the patch that passes your tests wins)"
   '

2. 2

   Run it on your keys (the model step, not checked by CI)

   Point your client at OrcaRouter; the hard_code rule fans out and tests_pass runs each candidate's patch against your suite, keeping the one that passes. The model runs and test execution are fenced.