Skip to content

About

The Quality Playbook is an open-source skill that turns AI coding agents into rigorous quality engineers. Point it at your code, in any language. Feed it whatever docs you have: READMEs, GitHub issues, Discord threads, AI chat history. It figures out your intent, creates a real requirements spec, and does quality engineering to find real defects.

Resources

Stars

90 stars

Watchers

1 watching

Forks

Latest commit

Β 

History

1,284 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Quality Playbook

Version: 1.6.1 | Author: Andrew Stellman | License: Apache 2.0

Find the bugs that code review misses

Most code review looks at how the code is written. The Quality Playbook looks at what the code is supposed to do, and whether it does it.

The Quality Playbook explores your codebase and works out the behavioral requirements of your code from two sources: the code itself, and your documentation (specs, issues, chat history, post-mortems). Then it runs a three-pass code review and a multi-model spec audit (the Council of Three) against those requirements. The bugs it turns up are the ones that look correct to anyone who doesn't already know the spec:

  • a function that silently returns null instead of throwing
  • a duplicate-key check that passes when the first value is null
  • sanitization that runs after the branch decision it was supposed to guard

It catches bugs that a thorough adversarial code-review prompt misses. The gson bug in the leaderboard below is one: the adversarial review of the same code, made without the playbook, is in that bug's evidence folder and doesn't report it. Along the way it also builds quality infrastructure that your project keeps, and every later review cycle runs against it:

  • derived requirements
  • functional and integration tests
  • contracts and a coverage matrix
  • the code-review, spec-audit, and TDD verification protocols themselves

And this isn't theoretical! Maintainers have merged or released fixes for 27 bugs that the Quality Playbook found in Javalin, the Linux kernel, Google's gson, aiohttp, calibre, OpenTelemetry and five other projects. Two of them now have CVE IDs: one credited only to the Quality Playbook's report, and one shared with independent reporters.

Quick start

Install into your project's root directory (pick one):

npx quality-playbook install --into . --ai-tool claude     # from npm, no global install
pip install quality-playbook && quality-playbook install --into . --ai-tool claude   # from pip
git clone https://github.com/andrewstellman/quality-playbook   # or clone and ask your AI to install it

Swap claude for cursor, copilot, continue, codex, windsurf, cline, or aider. You need Python 3.10 or later on your PATH.

Run by opening your project in your AI coding tool and telling the agent:

"Run the Quality Playbook on this project."

The agent finds the installed skill on its own and runs the six phases (explore, generate, code review, spec audit, reconcile, verify), pausing after each one until you say "keep going". It writes the results into a quality/ folder. Start with quality/BUGS.md, which lists the bugs it found.

Merged bugs leaderboard

As of 6 November 2026, maintainers have merged or released fixes for these 27 bugs, and 25 more fixes and reports are submitted and waiting. Three of them were private security reports, published by the maintainer as advisories once the fix was released, and GitHub has issued CVEs for the two addressable advisories: CVE-2026-126349 (GHSA-539f, credited only to my report) and CVE-2026-126350 (GHSA-j82w, credited to my report and two independent reporters). The full record, including the bugs that were held back, dropped or never filed, is in evidence/README.md.

Project The bug Fix Merged Evidence
Linux kernel, virtio-pci vp_interrupt() returned IRQ_NONE for a config-change interrupt with no vring work, even though reading the ISR had already consumed it, so the kernel counted those interrupts as spurious 93fa0945 2026-09-07 Evidence folder, with the QEMU console logs
Linux kernel, nvmet-rdma A wrapping offset check let nvmet_rdma_map_sgl_inline() write out of bounds (found independently; an earlier report by Bryam Vargas is credited) d1512773, queued for 7.3-stable v7.4-rc1 Evidence folder
Linux kernel, zram recompress_store() silently accepted an unrecognized type= value, leaving the pass with no slot filter so it recompressed every slot instead of the intended subset 2f529e73 2026-04-18 Evidence folder and the patch thread
Google gson gson silently accepted duplicate keys when the first value was null #3006 2026-04-05 Evidence folder
aio-libs aiohttp StreamReader.readuntil() missed a separator of two or more bytes when it was split across chunks, so it returned too much data or kept waiting #13870, backported to 3.15 and 3.14 2026-10-02 Evidence folder
calibre The content server returned a 500 instead of a 404 for a malformed hex id in an OPDS feed URL #3310 2026-09-29 Evidence folder
OpenTelemetry Java instrumentation The Forwarded host extractor recorded "[2001" as the host for a bracketed IPv6 address #20393 2026-10-16 Evidence folder
OpenTelemetry Java instrumentation UrlParser recorded server.address as "[" for a bracketed IPv6 address #20320 2026-10-16 Evidence folder
adonisjs http-server The uuid route matcher accepted letters that are not hex digits #144 2026-10-14 Evidence folder
python-poetry tomlkit Arithmetic on a float written with an explicit sign wrote a negative-zero result as +-0.0, which tomlkit itself then rejected #629 2026-10-12 Evidence folder
harrisiirak cron-parser String dates in the RFC 850 and asctime HTTP formats threw "unhandled timestamp", although cron-parser's README lists HTTP dates as supported #455 2026-10-11 Evidence folder
bradymholt cRonstrue A list with a plain first value and a step printed "At 09:00 AMonly on Friday and , every 2 days of the week" #423 2026-10-20 Evidence folder
sporkmonger addressable route_from returned an empty string when only the base URI had a query #623 2026-10-31 Evidence folder
sporkmonger addressable display_uri showed hosts containing bidi and invisible characters decoded from Punycode, so a host could be made to look like a different one (security advisory, low) GHSA-539f (CVE-2026-126349), fixed in 2.9.1 2026-10-30 Evidence folder
sporkmonger addressable Template#match still backtracked exponentially for {/var*}, {.var*} and {;var*} after the 2.9.0 ReDoS fix (security advisory, moderate; two other reporters found overlapping cases) GHSA-j82w (CVE-2026-126350), fixed in 2.9.1 2026-10-30 Evidence folder
harrisiirak cron-parser A crontab line or variable named __proto__ replaced the prototype of CronFileParser's errors and variables result objects instead of being stored as an entry (security advisory, low) GHSA-vh7f, fixed in 5.10.3 by #461 2026-10-31 Evidence folder
javalin A percent-encoded or double-slash path served a static file without running the path-scoped before-handler that guards it #2665, released in 7.2.4 2026-10-22 Evidence folder
javalin BasicAuthPlugin let every request through when no username and password were configured #2666, released in 7.2.4 2026-10-22 Evidence folder
javalin A HEAD request answered by a GET route was not checked against the GET route's roles #2668, released in 7.2.4 2026-10-22 Evidence folder
javalin HttpResponseException messages were returned unescaped as text/html #2669, released in 7.2.4 2026-10-22 Evidence folder
javalin A carriage return in a server-sent-events comment could start a new event field #2670, released in 7.2.4 2026-10-22 Evidence folder
javalin Basic auth credentials were ignored unless the scheme was spelled exactly Basic #2671, released in 7.2.4 2026-10-22 Evidence folder
javalin The lowercase-path redirect dropped the context path #2660, released in 7.2.4 2026-10-22 Evidence folder
javalin A static hostedPath was matched as a string prefix, so files were served at sibling URLs without their before-handler #2672, released in 7.2.4 2026-10-23 Evidence folder
javalin The precompression cache was shared across static-file configs, so one config's file could be served for another's #2673, released in 7.2.4 2026-10-23 Evidence folder
javalin Any request with a Sec-WebSocket-Version header created an HTTP session that never expired #2674, released in 7.2.4 2026-10-23 Evidence folder
javalin UploadedFile.filename() kept directory components such as ../ from the client #2675, released in 7.2.4 2026-10-23 Evidence folder

Open-source maintainers are drowning in AI-generated pull requests right now, and I have no intention of adding to that pile. The playbook's agents reproduced each of these bugs and drafted the patches, but before anything went upstream I read the code, checked that each regression test failed without the fix and passed with it, and checked the output against the finding. For the virtio bug that meant booting a patched kernel under QEMU. Every one of these fixes says in its pull request or commit message that AI was involved, except the zram patch, which went in before I started requiring that.

Every bug in this table and the next has a record in this repository, which the evidence column links to: the reproduction logs, the patch as sent, and (for all but virtio, zram and gson, which came before I added that step) the review that preceded submission. The gson and zram records are the raw run output, recovered from a backup after the fact; each folder's NOTE.md says what is and is not there.

Contents

How to install the Quality Playbook

The fastest way to install is from npm or pip. From your project's root directory, run one of these:

# From npm, no global install:
npx quality-playbook install --into . --ai-tool <tool>

# From pip / uvx / pipx (Python 3.10+):
uvx quality-playbook install --into . --ai-tool <tool>           # one-shot, no global install
pipx run quality-playbook install --into . --ai-tool <tool>
pip install quality-playbook && quality-playbook install --into . --ai-tool <tool>

<tool> is one of claude, cursor, copilot, continue, codex, windsurf, cline, or aider. The skill installs into .<tool>/skills/quality-playbook/ (or .github/skills/quality-playbook/ for copilot). These commands install it for Claude Code, Cursor and GitHub Copilot:

npx quality-playbook install --into . --ai-tool claude       # Claude Code
npx quality-playbook install --into . --ai-tool cursor       # Cursor
npx quality-playbook install --into . --ai-tool copilot      # GitHub Copilot

You can also install the Quality Playbook (QPB) from the Claude Code plugin marketplace. If you use Claude Code, the first command below adds the marketplace and the second installs QPB from it:

/plugin marketplace add https://github.com/andrewstellman/quality-playbook
/plugin install quality-playbook

Use the full https:// URL, because the shorthand github.com/... form makes Claude Code attempt an SSH clone, which fails on machines without a GitHub SSH key configured.

After you install it this way, the agent finds the QPB skill in any project you open with Claude Code, so you don't need to run quality-playbook install --into .... When a new QPB version ships, run /plugin marketplace update and then install it again.

To work from a local checkout of QPB (for example, to test a fork), use --plugin-dir instead of the marketplace:

claude --plugin-dir /path/to/quality-playbook/plugins/quality-playbook

The --plugin-dir argument takes an absolute path to the plugin directory inside the QPB clone, and it loads the plugin for that Claude Code session only.

You can also ask your AI coding tool to install it from a clone of this repo.

  1. Clone this repo somewhere on your machine (for example, git clone https://github.com/andrewstellman/quality-playbook ~/quality-playbook). You can install into any number of projects from one clone.

  2. Open your target project in Claude Code, Cursor, GitHub Copilot, Windsurf, Continue, or another AI coding tool.

  3. Ask the AI to install it. You can say something like this:

    "Install the Quality Playbook into this project from ~/quality-playbook."

    The agent reads AGENTS.md, figures out which install location your tool uses, and runs the installer.

If you'd rather install by hand or run the script yourself, load ai_context/TOOLKIT.md into your AI tool (there's more about that file under Need help? below) and ask it: "Read TOOLKIT.md. How do I install the skill manually or via the bin/install_skill.py script directly?"

Prerequisite: You need Python 3.10 or later on your PATH. The npm package is a thin shim over the same Python installer, so npx needs Python 3.10 too.

The playbook finds more bugs when you give it documentation. It reads written specs, design docs, GitHub or Jira issues from real users, chat history, and post-mortems, and from those it works out what your code is supposed to do. Without documentation it still runs from the source tree alone, but it finds fewer bugs. Put everything you have into a reference_docs/ directory at the project root.

One prompt gathers the documentation for you. Copy references/DOC_GATHERING_PROMPT.md into your AI coding tool with your project open, and run it. The agent confirms which project you mean, then collects the project's docs, issues, and advisories into reference_docs/.

How to run the Quality Playbook

Open your project in your AI coding tool (Claude Code, Cursor, GitHub Copilot, Windsurf, Continue, etc.) and tell the agent:

"Run the Quality Playbook on this project."

That one line is enough, because once the skill is installed the agent finds it on its own; you don't have to open or point at SKILL.md or any other file. The agent runs the six phases, pausing after each one until you say "keep going", and writes the results into a quality/ folder in your project.

A full six-phase run takes a while and uses a lot of tokens. To spread it across sessions, for example to stay inside a daily token budget, tell the agent to run only some of the phases:

"Run phases 1 to 3 of the Quality Playbook on this project."

Then, in a later session:

"Continue the Quality Playbook from phase 4."

When the run finishes, the quality/ folder contains:

quality/
β”œβ”€β”€ BUGS.md                  ← consolidated bug report with spec basis (start here)
β”œβ”€β”€ REQUIREMENTS.md          ← behavioral requirements derived from your code + docs
β”œβ”€β”€ EXPLORATION.md           ← Phase 1 findings: patterns explored, files tagged
β”œβ”€β”€ QUALITY.md               ← quality constitution for your codebase
β”œβ”€β”€ CONTRACTS.md             ← extracted behavioral contracts
β”œβ”€β”€ COVERAGE_MATRIX.md       ← contract-to-requirement traceability
β”œβ”€β”€ COMPLETENESS_REPORT.md   ← final gate report with post-reconciliation verdict
β”œβ”€β”€ PROGRESS.md              ← phase checkpoint log + cumulative bug tracker
β”œβ”€β”€ test_functional.py       ← functional tests traced to requirements
β”œβ”€β”€ test_regression.py       ← regression tests for confirmed bugs
β”œβ”€β”€ writeups/                ← per-bug detailed writeups with patches (BUG-NNN.md)
β”œβ”€β”€ patches/                 ← fix and regression-test patches
β”œβ”€β”€ code_reviews/            ← three-pass code review output
β”œβ”€β”€ spec_audits/             ← Council of Three auditor reports + triage
└── results/                 ← TDD red/green logs, integration results, gate log

Start with BUGS.md for the headline findings. Then read REQUIREMENTS.md, the playbook's account of what your code is supposed to do, which includes requirements it derived from issues and docs you may not have known about. The bugs it reports come from the gap between what REQUIREMENTS.md says and what your code does.

Need help? Just ask your AI

The rest of this README covers the playbook's phases and output files, but the easiest way to get answers is to download one file, attach it to your favorite AI chatbot, and ask it whatever you want to know.

The file is ai_context/TOOLKIT.md. It's a single Markdown document that explains the Quality Playbook, written so that an AI assistant can read it and answer your questions about the playbook.

Open a chat in whatever AI tool you use, attach TOOLKIT.md, and tell it:

"Read TOOLKIT.md. Now you're an expert in the Quality Playbook."

ChatGPT with TOOLKIT.md attached

Then ask it anything, from how to set the playbook up to why your run found only one bug. The Want to learn more? section below has example prompts.

This ChatGPT conversation shows what that looks like, and it works the same way in other AI tools.

Running the playbook

The playbook runs in six phases, each in its own context window. After each phase, say "keep going" to continue.

Phase Purpose
1. Explore The agent reads the code, tests, config, specs, commit history, and reference_docs/ if it's there, to work out both what the code does and what it's supposed to do.
2. Generate A five-stage pipeline derives the behavioral requirements, then generates the functional and integration tests, the code-review, spec-audit and TDD protocols, and the quality constitution.
3. Code review Three-pass review: structural, requirement verification, cross-requirement consistency. Each confirmed bug gets a regression test.
4. Spec audit Three independent AI models audit the code against the requirements (the Council of Three). Triage uses verification probes ("is this actually true?"), and the models don't vote.
5. Reconciliation Every bug from the code review and the spec audit is tracked, and either gets a regression test or is explicitly exempted.
6. Verify 45 self-check benchmarks validate generated artifacts. Final gate: GATE PASSED / GATE PASSED WITH CLEANUP NEEDED / GATE PASSED WITH DECISIONS NEEDED / GATE FAILED.

After Phase 6 the agent presents a summary of the results and a menu of next steps (SKILL.md calls this Phase 7). Since v1.6.1 it can also prepare an upstream submission for one confirmed bug at a time: a policy check, confirmation, a fix, a blind review panel and a draft PR. It never submits anything itself.

After the first six-phase run (the baseline), you can run four iteration strategies in order: gap, unfiltered, parity, and adversarial. Each strategy looks for different classes of bug, and running all four typically adds 40 to 60 percent more confirmed bugs on top of the baseline. Say "Run the next iteration using the gap strategy" to begin.

After you fix bugs, say "recheck". Recheck mode checks your fixes against the existing bug report without re-running the full pipeline, and takes 2 to 10 minutes.

To run the playbook autonomously or in CI, use the prompt under Running autonomously or in CI in Want to learn more? below, or the orchestrator agent in plugins/quality-playbook/skills/quality-playbook/agents/quality-playbook.agent.md.

Run time and token use

A full baseline run takes hours and uses a large share of a daily or weekly quota on most providers. One measured example: the run on Javalin (about 13,000 lines of non-test Kotlin and Java) on 29 and 30 September 2026, with QPB 1.6.0 on Claude Opus 5.5, took about 24 hours from start to finish and used the equivalent of about $94 at API prices. That run found the Javalin bugs in the leaderboard above.

For multi-repo benchmark work, see harness_plans/ for parallel-pool plan formats and docs/design/QPB_Test_Harness_*.md for the harness internals.

Known limitations

The phase checks rely on the agent following written instructions; the code doesn't enforce them. Phases 1, 2, and 5 tell the agent to run validate_phase_artifacts and quote its verdict line, and Phase 6 tells it to use a fresh-context auditor sub-agent. In one Phase 1 run, Codex desktop reported PASS without running the validator. When you review a phase verdict, check for the verbatim RESULT: VALIDATION PASSED (phase N) line, or for fresh-context framing in the Phase 6 auditor verdict.

The proposal for enforcing these checks in code is in docs/design/QPB_v1.6.x_Phase6_Structural_Enforcement_Proposal.md.

What the playbook produces

The playbook generates these files:

Artifact Location What it does
REQUIREMENTS.md quality/ Behavioral requirements derived from code, docs, and community sources via a five-stage pipeline. Without requirements, review can find only structural bugs.
QUALITY.md quality/ Quality constitution defining what "correct" means for this specific project, with fitness-to-purpose scenarios and rules against coverage theater (tests that look thorough but aren't).
test_functional.* quality/ Functional tests in the project's native language, traced to requirements rather than generated from source code.
RUN_CODE_REVIEW.md quality/ Three-pass protocol: structural review, requirement verification, cross-requirement consistency. Each pass is designed to find bugs the other two miss.
RUN_SPEC_AUDIT.md quality/ Council of Three: three independent AI models audit the code against requirements. Different models have different blind spots, and the triage uses verification probes (targeted checks that ask "is this actually true?"), and the models don't vote.
RUN_INTEGRATION_TESTS.md quality/ End-to-end test protocol grounded in use cases, with a traceability column mapping each test to the user outcome it validates.
RUN_TDD_TESTS.md quality/ Red-green TDD verification protocol that, for each confirmed bug, shows the regression test failing on unpatched code and passing with the fix.
BUGS.md quality/ Consolidated bug report with spec basis, severity, reproduction steps, and patch references for every confirmed finding.
AGENTS.md project root Bootstrap file so every future AI session inherits the full quality infrastructure.

How it works

The playbook's value comes from requirement derivation. AI code reviewers have the same limit human reviewers do. If you don't know what the code is supposed to do, you can only find structural issues. The playbook's main job is working out that intent from your code and your documentation, and then building every later artifact from it.

For more detail on any of the phases in the Running the playbook table above, ask your AI tool:

"Read TOOLKIT.md. Walk me through Phase N in detail. What's the input, what's the output, what can go wrong?"

The final gate produces one of four verdicts:

  • GATE PASSED: the review is complete and there is nothing to do.
  • GATE PASSED WITH CLEANUP NEEDED: the bug findings are real and stand on their own, and only the audit trail is incomplete.
  • GATE PASSED WITH DECISIONS NEEDED: there is no substantive problem, but some requirements claim more than the passage they cite. No confirmed bug rests on those requirements, and quality/OPERATOR_DECISIONS.md lists your options for each one.
  • GATE FAILED: a substantive problem, such as an incomplete review, missing specs, or a fabricated verdict.

The four verdicts separate "your code is broken in N ways" from "your audit trail is incomplete in N ways". Every verdict starts with a ── Bug evidence ── block that says how many bugs have red and green test logs; that count is reported on its own, apart from the pass or fail.

Why documentation matters

In a controlled experiment across multiple repositories, runs with documentation found more bugs than the code-only runs, and different ones, and reported them with higher confidence. The documentation gives the auditors spec language to check against, so "this code looks odd" becomes "this code contradicts the documented behavior".

Want to learn more?

You can answer most questions by loading TOOLKIT.md into your AI tool and asking. These example prompts cover common situations:

Setting up documentation:

"Read TOOLKIT.md. What documentation should I provide and how do I gather it? What's the difference between top-level reference_docs/ and reference_docs/cite/? What about projects with no documentation?"

Understanding what each phase does:

"Read TOOLKIT.md. What does Phase 3 actually do? How does it find bugs that structural code review misses? How is it different from Phase 4?"

Choosing iteration strategies:

"Read TOOLKIT.md. What are the four iteration strategies (gap, unfiltered, parity, adversarial)? When should I use each? What's the right order?"

Running autonomously or in CI:

"Read TOOLKIT.md. How do I run the playbook autonomously across multiple iterations without babysitting it? What about CI?"

Interpreting low bug counts:

"Read TOOLKIT.md. My run only found one bug. Is my code that clean, or did the playbook miss something? How do I tell?"

The Council of Three:

"Read TOOLKIT.md. What is the Council of Three? Why three models? How does the triage handle disagreements?"

Manual install fallback:

"Read TOOLKIT.md. The automatic install didn't work. How do I copy the skill files manually for Claude Code, Cursor, GitHub Copilot, Continue, Windsurf, Codex, Cline, or Aider?"

Tuning recall:

"Read TOOLKIT.md. My playbook missed a bug I know is in the code. What can I tune on the next run?"

Understanding the gate verdicts:

"Read TOOLKIT.md. What's the difference between GATE PASSED, GATE PASSED WITH CLEANUP NEEDED, GATE PASSED WITH DECISIONS NEEDED, and GATE FAILED?"

If TOOLKIT.md doesn't answer your question, file an issue at https://github.com/andrewstellman/quality-playbook/issues.

Submitted upstream

These fixes and reports are still waiting on the maintainers. All of them except those opened in November (setuptools #5374 and #5375, zod #6767, #6769 and #6770, tomlkit #714, addressable #702 and #703) were checked on GitHub on 27 October 2026, and none had been merged or closed. The Linux patches were checked against the history of mainline (torvalds/linux), not their mailing-list threads. The five marshmallow pull requests from #3086 to #3090 share one evidence folder: three of them were first reported privately to marshmallow's security contact, who ruled them not security issues and asked for public pull requests. Reports that may be security issues and are still private are not listed until the maintainers publish them.

Project The bug Submitted Evidence
wanasit chrono With a reference timezone that differs from the system's, "this month", "last night" and dates without a year used the system timezone's calendar #670 Evidence folder
RubyMoney money Dividing a subclass of Money by a plain Money returned a Money instead of a Float #1237 (approved by a maintainer, with a suggested extra test) Evidence folder
javalin enableBasicAuth did not protect WebSocket upgrades #2667 (held by the maintainer for Javalin 8, where it changes documented behaviour) Evidence folder
marshmallow URL(relative=True) rejected a relative URL with "://" in its path, query or fragment #3067 Evidence folder
marshmallow A pre_load(pass_collection=True) hook that changes the data's length made a per-item @validates_schema raise ValueError under many=True #3086 Evidence folder
marshmallow A @validates method never ran for a field with a dotted attribute #3087 Evidence folder
marshmallow A value rejected by @validates on a field with attribute set stayed in ValidationError.valid_data #3088 Evidence folder
marshmallow Nested dropped dotted load_only and dump_only for a nested schema instance #3089 Evidence folder
marshmallow Under unknown=INCLUDE, an unknown key that matched a loaded field's key overwrote the loaded value #3090 Evidence folder
pypa setuptools readme or requires-python set outside dynamic crashed instead of being warned about and ignored #5343 (the maintainer has pushed review changes to it) Evidence folder
pypa setuptools MANIFEST.in exclude patterns ignored character ranges such as [a-c] #5374 (an outside reviewer caught a reversed-range edge case, fixed in a follow-up commit) Evidence folder
pypa setuptools exclude_package_data didn't exclude a file whose name is stored in NFD, as macOS does, when the pattern is NFC (first reported privately to setuptools' Tidelift contact, who said to make it public) #5375 Evidence folder
spf13 cobra Command suggestions measured edit distance in bytes, not characters #2514 Evidence folder
colinhacks zod registry.remove() deleted an id that now belonged to another schema issue #6640 (zod accepts pull requests only from collaborators) Evidence folder
colinhacks zod An intersection threw when both sides produced NaN issue #6639 Evidence folder
colinhacks zod Compiled parsing accepted a missing key whose optional was inside z.intersection() or z.success() issue #6767 (fix on a branch of my fork) Evidence folder
colinhacks zod An exhaustive z.record() accepted a missing key when the value schema accepted undefined issue #6769 (fix on a branch of my fork) Evidence folder
colinhacks zod A constant .catch() value and a Date default were shared across parses issue #6770 (fix on a branch of my fork) Evidence folder
python-poetry tomlkit parse() raised RecursionError on a valid document whose dotted key was within the nesting limit issue #714 Evidence folder
sporkmonger addressable normalize decoded %3A, %5B and %5D in a host name, so an encoded colon became a port #702 Evidence folder
sporkmonger addressable The pure-Ruby to_ascii skipped the characters nameprep removes, such as a zero width joiner #703 Evidence folder
NVIDIA bionemo-recipes The THD context-parallel split dropped the remainder tokens #1765 Evidence folder
NVIDIA bionemo-recipes AMPLIFY _pad_weights returned fp32 for bf16 input #1764 Evidence folder
Linux kernel nvmet ANA group ID 128 was clamped to 0 by array_index_nospec() v2 applied to nvme-7.4 by Keith Busch, 6 November 2026, not yet in mainline Evidence folder
Linux kernel nvmet The CRTO property was computed from CSTS instead of CAP patch sent to linux-nvme, 10 September 2026 Evidence folder

Recent releases

These are the highlights; CHANGELOG.md has the detail for every release.

  • v1.6.1: the release built from the open-source bug campaign. Sub-agents under the parent-witness rule: phases may run in per-phase sub-agents, as long as the parent session that spawned them re-runs the gate itself after Phase 6 (scripts/qpb_gate_witness.py in the plugin's skill folder). Preparing upstream submissions: a new path after the six phases takes one confirmed bug at a time through a policy check, confirmation, a fix, a blind review panel and a prepared draft PR. The skill never submits anything upstream itself; five of its guardrails are written into SKILL.md and the rest are in references/upstream_submission.md. The release also adds rules for the claims QPB makes in its own output, a gate verdict that reports decisions and bug evidence plainly, and fixes to the persona validation that v1.6.0 introduced.
  • v1.6.0: the requirements review and grounding release. Dump-and-go documentation classification: drop any docs in reference_docs/ and the model reads and classifies each one. The model cites authoritative references (marked unconfirmed until you confirm them), leaves CVEs, advisories and source code for you to decide on, and doesn't demote a real reference just because the project's maintainers didn't write it. Agent-persona requirements validation: fresh-context domain-expert and security personas independently check the derived requirements and correct them, with every change grounded in a document and revertable. The release also adds a mechanically checked structure for the requirements document, an opt-in interview in which you validate the requirements, and durable evidence of what you confirmed.
  • v1.5.10: repo hygiene. The canonical SKILL.md and references/ moved back to the repo root as the single source of truth (the in-tree skill locations are symlinks), SKILL.md was trimmed, with per-phase detail loaded from references/*.md as needed, and run output and orphaned partial copies that had been committed were removed from tracking.
  • v1.5.8: distribution channels. QPB was published to pip (PyPI), npm, and the Claude Code plugin marketplace, with publish scripts that require an explicit --publish after a --dry-run (and --otp for 2FA), and the repository was restructured to Claude Code's standard self-hosted plugin marketplace layout (plugins/quality-playbook/).
  • v1.5.7: channel scaffolding for pip, uvx, pipx and npx, phase-aware bundling, the Council of Three review protocol was formalized, and the worker self-Council pattern (parallel sub-agent reviewers through the Task tool) was added.
  • v1.5.6: the improvement loop formalized. The v1.5.x releases build the quality-control infrastructure (finding bugs, validating the skill's prose), and v1.6 onward builds quality improvement on top of it (statistical control, multi-operator workflows).

Running across many repos with arunner

For multi-repo runs, benchmark suites, or unattended batch work, drive the playbook with arunner, a batch orchestrator for agentic coding systems, written with only the Python standard library. You point it at a list of jobs (run the playbook across these ten repos, audit these branches), and it runs them in a pool, watches each job's heartbeat, and leaves a complete record on disk. It needs no server, no daemon, and no API key beyond your existing agent session, and to recover from a crash you run one tick (one pass of its scheduler, which is idempotent): if you close the window or the machine goes to sleep, arunner picks up where it left off.

arunner is vendor-neutral because its orchestration engine is plain Python and a job is anything that appends JSON lines to a file, so a QPB run, a shell script, or a CI job all qualify. I recommend it for running QPB across a set of repos without babysitting each one.

See the arunner repository for the worker contract, plan formats, and host support table. Native support for running QPB's phases through arunner is still in development. Until it ships, run QPB through arunner as a shell or agent job, or run it phase by phase as described in How to run the Quality Playbook above.

Repository structure

The canonical SKILL.md and its references/ directory are at the repo root, and the skill folders inside the plugin tree are symlinks back to them. QPB uses Claude Code's standard self-hosted marketplace layout: .claude-plugin/marketplace.json points at the plugin under plugins/quality-playbook/. The pip and npm packages copy the real files from the root, with the symlinks resolved.

quality-playbook/
β”œβ”€β”€ SKILL.md                 # The skill: canonical source of truth (full operational instructions)
β”œβ”€β”€ references/              # Protocol + per-phase pipeline reference docs (lazy-loaded per phase)
β”‚   β”œβ”€β”€ DOC_GATHERING_PROMPT.md
β”‚   β”œβ”€β”€ exploration_patterns.md
β”‚   β”œβ”€β”€ requirements_pipeline.md
β”‚   β”œβ”€β”€ review_protocols.md
β”‚   β”œβ”€β”€ spec_audit.md
β”‚   β”œβ”€β”€ verification.md
β”‚   β”œβ”€β”€ phase1_exploration_guide.md
β”‚   β”œβ”€β”€ phase2_generation_guide.md
β”‚   β”œβ”€β”€ phase5_reconciliation_guide.md
β”‚   β”œβ”€β”€ phase6_verify_guide.md
β”‚   β”œβ”€β”€ phase7_guide.md
β”‚   β”œβ”€β”€ recheck_mode.md
β”‚   β”œβ”€β”€ run_state_schema.md
β”‚   └── ... (other reference docs)
β”œβ”€β”€ schemas.md               # Artifact schema definitions
β”œβ”€β”€ AGENTS.md                # AI bootstrap file (repo root)
β”œβ”€β”€ .claude-plugin/          # ROOT-only: marketplace catalog
β”‚   └── marketplace.json     # Self-hosted marketplace entry (source: ./plugins/quality-playbook)
β”œβ”€β”€ plugins/                 # Plugin tree (standard self-hosted marketplace layout)
β”‚   └── quality-playbook/    # The plugin itself
β”‚       β”œβ”€β”€ .claude-plugin/
β”‚       β”‚   └── plugin.json  # Plugin metadata (name, description, version, author)
β”‚       └── skills/
β”‚           └── quality-playbook/
β”‚               β”œβ”€β”€ SKILL.md      # Symlink β†’ repo-root SKILL.md
β”‚               β”œβ”€β”€ references/   # Symlink β†’ repo-root references/
β”‚               β”œβ”€β”€ phase_prompts/   # Per-phase agent prompts (Mode A + Mode B)
β”‚               β”œβ”€β”€ agents/          # Orchestrator agent files for autonomous runs
β”‚               β”œβ”€β”€ ai_context/      # Adopter-facing AI context (TOOLKIT.md symlink)
β”‚               └── scripts/         # Bundled scripts (quality_gate.py, install_skill.py, ...)
β”œβ”€β”€ bin/                     # Repo-level runner + build scripts (Python 3.10+)
β”‚   β”œβ”€β”€ run_playbook.py      # Mode B runner (positional args are target directories)
β”‚   β”œβ”€β”€ build_channel_package.py  # Stages the pip/npm bundle (dereferences symlinks)
β”‚   β”œβ”€β”€ publish_pip.py       # Pip publish path
β”‚   β”œβ”€β”€ publish_npm.py       # Npm publish path
β”‚   β”œβ”€β”€ install_skill.py     # Thin shim that delegates to the plugin's scripts/install_skill.py
β”‚   └── tests/               # stdlib-only unit tests (python3 -m pytest bin/tests/)
β”œβ”€β”€ pytest/                  # Local stdlib-only shim (python3 -m pytest works without installs)
β”œβ”€β”€ ai_context/              # Maintainer orientation docs + adopter-facing TOOLKIT.md
β”‚   β”œβ”€β”€ DEVELOPMENT_CONTEXT.md
β”‚   β”œβ”€β”€ DEVELOPMENT_PROCESS.md
β”‚   β”œβ”€β”€ IMPROVEMENT_LOOP.md
β”‚   β”œβ”€β”€ TOOLKIT_TEST_PROTOCOL.md
β”‚   β”œβ”€β”€ BENCHMARK_PROTOCOL.md
β”‚   └── TOOLKIT.md           # Adopter-facing AI context; symlinked into the skill bundle
β”œβ”€β”€ harness_plans/           # Parallel-pool benchmark plan formats
β”œβ”€β”€ runner/                  # arunner run directories (per-version harness state)
β”œβ”€β”€ docs/                    # Design docs, implementation plans, proposals
β”œβ”€β”€ reviews/                 # Council-of-Three review artifacts
β”œβ”€β”€ metrics/                 # Cross-version benchmark metrics
β”œβ”€β”€ images/                  # README images
β”œβ”€β”€ reference_docs/          # Documentation QPB reads to derive requirements
β”œβ”€β”€ scripts/                 # Repo-level helper scripts
β”œβ”€β”€ quality_playbook_cli/    # CLI package
β”œβ”€β”€ quality/                 # quality/audits/ only; the self-run output was removed from tracking in v1.5.10
β”œβ”€β”€ pyproject.toml           # Pip packaging + bundle globs
β”œβ”€β”€ package.json             # npm shim packaging
β”œβ”€β”€ CHANGELOG.md             # Per-release detail
└── LICENSE.txt              # Apache 2.0

How I improve the playbook

Each release goes through a Plan-Do-Check-Act loop, and the Check step is benchmark recovery against pinned ground truth. I make a change and run the playbook against three pinned benchmark repositories (chi-1.5.1, virtio-1.5.1, express-1.5.1) whose bug counts are already known from earlier releases. The release ships only if both verification dimensions, process compliance and outcome recall, hold or improve.

Verification dimensions are what I measure on every release: process compliance (does the run produce the right artifacts?) and outcome recall (does the run find the bugs I know are there?). The most dangerous failure is a run that passes on process but fails on recall, which means every gate is green while the run found no real bugs.

Improvement levers are what I change to make the playbook better. Each lever lives in a known place in the codebase and can be tuned without affecting the others. The current levers are:

  • exploration breadth and depth (references/exploration_patterns.md, references/iteration.md)
  • code-derived versus domain-derived requirements (references/requirements_*.md plus the skill's scripts/citation_verifier.py)
  • gate strictness (the skill's scripts/quality_gate.py)
  • finalization robustness (bin/run_playbook.py::_finalize_iteration)
  • the mechanical citation extractor (bin/skill_derivation/citation_search.py, with its token-overlap pre-filter)
  • the four-pass skill-derivation pipeline (bin/skill_derivation/pass_{a,b,c,d}.py plus the divergence-detection modules under bin/skill_derivation/divergence_*.py)

I connect the levers to outcome recall with regression replay. I take a pinned benchmark, roll it back to a commit just before a known bug was fixed, and run the playbook against that pre-fix commit. If the playbook finds the bug, the levers are good enough for that class of bug. If it misses the bug, I work out which lever to change, change it, and re-run the playbook to check both that it now finds the bug and that recall on the rest of the benchmark holds.

ai_context/IMPROVEMENT_LOOP.md has the full detail, including the lever inventory with file mappings and the plan for moving to formal statistical process control. The review that checks the ai_context/ orientation docs before each release (the documentation version of the Council of Three) is described in ai_context/TOOLKIT_TEST_PROTOCOL.md.

Context

This project accompanies the O'Reilly Radar article AI Is Writing Our Code Faster Than We Can Verify It, part of my series on AI-driven development for O'Reilly Radar. I built the playbook using AI-driven development, with Octobatch, an open-source Python batch LLM orchestrator. This README was coauthored with Claude.

License

Apache 2.0.

Patent notice

Aspects of the methodology described in this repository are the subject of US Provisional Patent Application No. 64/044,178, filed April 20, 2026 by Andrew Stellman.

Users of this project are covered by the Apache License 2.0, which includes an express patent grant in Section 3. That grant is perpetual, worldwide, royalty-free, and irrevocable (except as described in the license), and extends to anyone using, reproducing, modifying, or distributing the Quality Playbook under the terms of the Apache 2.0 license. Nothing in this notice diminishes that grant.

The patent application exists to preserve a defensive priority date; it is not asserted against users, contributors, forks, or derivative works of this project practiced under Apache 2.0.

About

The Quality Playbook is an open-source skill that turns AI coding agents into rigorous quality engineers. Point it at your code, in any language. Feed it whatever docs you have: READMEs, GitHub issues, Discord threads, AI chat history. It figures out your intent, creates a real requirements spec, and does quality engineering to find real defects.

Resources

Stars

90 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages