promptfoo: make agent evals fail the build, not the user

The Stack

Steps

1. 1

   Author the eval config + CI workflow and validate them

   Write promptfooconfig.yaml (providers, prompts, tests with assertions) and a CI workflow that runs promptfoo eval on every push. CI parses the YAML and asserts providers + tests-with-assertions exist and that CI runs the evals. The eval scoring a live model is fenced.

   mkdir -p .github/workflows
   cat > promptfooconfig.yaml <<'YAML'
   description: "Agent behavior eval"
   providers:
     - anthropic:messages:claude-opus-4.8
   prompts:
     - "Answer concisely: {{q}}"
   tests:
     - vars: { q: "What is 2 + 2?" }
       assert:
         - type: contains
           value: "4"
     - vars: { q: "What is the capital of France?" }
       assert:
         - type: icontains
           value: "Paris"
   YAML
   cat > .github/workflows/evals.yml <<'YML'
   name: evals
   on: [push]
   jobs:
     promptfoo:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
         - run: npx promptfoo@latest eval -c promptfooconfig.yaml
   YML
   ruby -ryaml -e '
   c = YAML.load_file("promptfooconfig.yaml")
   providers = c["providers"]
   abort "BAD: no providers" unless providers.is_a?(Array) && providers.length >= 1
   tests = c["tests"]
   abort "BAD: no tests" unless tests.is_a?(Array) && tests.length >= 1
   with_assert = tests.count { |t| t["assert"].is_a?(Array) && t["assert"].length >= 1 }
   abort "BAD: tests carry no assertions" unless with_assert >= 1
   wf = File.read(".github/workflows/evals.yml")
   abort "BAD: CI does not run promptfoo eval" unless wf.include?("promptfoo") && wf.include?("eval")
   puts "config OK: promptfoo config parses with " + providers.length.to_s + " provider(s) and " + with_assert.to_s + " test(s) carrying assertions; CI runs promptfoo eval"
   '

2. 2

   Run the evals (the model step, not checked by CI)

   Set your provider keys and run npx promptfoo eval locally or in CI. Because model-backed evals are noisy, lean on concrete assertions and run repeats. The scoring is fenced.