CodingHybridFreeActiveMachine-verified· beginner · ~5 min setup

agent-qa: validate Agent Skills before LLM-backed QA runs

Run agent-qa's deterministic skill-pack validator before its authoring, result-triage, or debug-fix skills can drive a model-backed web or mobile QA run, so missing metadata, references, or MCP guidance fail in CI first.

by Shilpa Mitra· verified today· v1.0.0

Run this workflow

CI-verified, 2/2 fixtures passing.

Build this with your agent

One copy-paste hands Claude Code, Codex, or Cursor the full recipe, steps included, nothing to fetch.

Intended Use

Teams vendoring or checking out agent-qa who want a cheap CI gate before Codex or Claude Code uses its authoring, result-triage, or debug-fix skills. CI fetches the immutable reviewed commit 22636e1, runs the upstream validator without inherited environment variables, and asserts its exact success message plus exit code. No API key and no model call.

Not for

  • Claiming the three skills are behaviorally correct, the validator checks files and text markers but does not judge whether the instructions produce good tests, diagnoses, or fixes
  • Proving MCP connectivity or workspace permissions, agents/openai.yaml only has to exist; the validator does not parse it or call the local agent-qa MCP server
  • Proving a web, Android, or iOS run succeeds, real runs need browser or Appium setup plus a multimodal model and remain nondeterministic
  • Treating agent-qa as OSI open source today, its FSL license restricts competing commercial use until each version converts to Apache-2.0 after two years

The Stack

Agent Skillsskill packaging formatagent-qaagentic QA harness and shipped skill pack

Tested Against

vostride/agent-qa@22636e183bb4ee4528bf90db676837369a498e8fagent-qa skills validator (2026-08-03)node@24

Side effects & data flow

Network
codeload.github.com
Writes
./agent-qa-22636e183bb4ee4528bf90db676837369a498e8f/, ./agent-qa.tar.gz, ./.validator-home/
Credentials
none required

Data privacy

  • GitHub ← HTTPS request for a public, pinned agent-qa source archive; no project or test data (retention: per GitHub's privacy statement)

Prerequisites

  • Node.js 24+ (the repository's declared engine)
  • An agent-qa checkout when following the command outside FlowStacks CI

Steps

  1. 1

    Fetch the reviewed skill pack and run its keyless validator

    Pin the source rather than following main, then run the repository's own validator before any model-backed authoring, triage, or debug/fix work. FlowStacks CI downloads commit 22636e1 and executes the reviewed validator with an empty environment so no CI credentials are inherited. The exact success line and exit code are graded.

    curl --proto '=https' --tlsv1.2 --retry 3 -fsSL   https://codeload.github.com/vostride/agent-qa/tar.gz/22636e183bb4ee4528bf90db676837369a498e8f   -o agent-qa.tar.gz
tar -xzf agent-qa.tar.gz
mkdir -p .validator-home
cd agent-qa-22636e183bb4ee4528bf90db676837369a498e8f
env -i PATH="$PATH" HOME="$PWD/../.validator-home" node skills/scripts/validate-skills.mjs
  2. 2

    Only after the gate passes, use the skill that matches the job

    Use agent-qa-authoring to create or validate tests with generated canonical IDs; agent-qa-result-triage to classify a failed run from concrete artifacts and logs; and agent-qa-debug-fix only after triage to make the smallest evidence-backed patch. Connecting the MCP server and running a real multimodal test are fenced because they require a live project, browser or device runtime, and model access.

Eval, 2 fixtures

Last passed: verified today
  • skills-pack-okcontainstimeout 120s · max $0

    Expected: agent-qa skills pack validation passed

  • clean-exitexit_codetimeout 120s · max $0

    Expected: 0

Results

agent-qa ships three Agent Skills: one authors and validates tests with canonical IDs, one triages failed runs from artifacts, and one patches the smallest evidence-backed fix. Its keyless Node validator checks that every skill has SKILL.md and agents/openai.yaml, that the skill text contains frontmatter name/description markers and current agent_qa_ MCP guidance (with the stale agentqa_ prefix rejected), and that the authoring and triage reference files exist. That is a useful packaging gate, not an end-to-end QA test: it does not parse the YAML files, prove the MCP server connects, judge the instructions, or run a model against a browser or device.

Did this work for you?

Our CI checks the setup runs. You tell us if the whole thing worked. Tell us straight.

Related workflows

Liked this workflow?

Get new verified workflows in WebAfterAI, three issues a week (Tue, Thu, Sat).