Version: 1.6.1 | Author: Andrew Stellman | License: Apache 2.0
Find the bugs that code review misses
Most code review looks at how the code is written. The Quality Playbook looks at what the code is supposed to do, and whether it does it.
The Quality Playbook explores your codebase and works out the behavioral requirements of your code from two sources: the code itself, and your documentation (specs, issues, chat history, post-mortems). Then it runs a three-pass code review and a multi-model spec audit (the Council of Three) against those requirements. The bugs it turns up are the ones that look correct to anyone who doesn't already know the spec:
- a function that silently returns null instead of throwing
- a duplicate-key check that passes when the first value is null
- sanitization that runs after the branch decision it was supposed to guard
It catches bugs that a thorough adversarial code-review prompt misses. The gson bug in the leaderboard below is one: the adversarial review of the same code, made without the playbook, is in that bug's evidence folder and doesn't report it. Along the way it also builds quality infrastructure that your project keeps, and every later review cycle runs against it:
- derived requirements
- functional and integration tests
- contracts and a coverage matrix
- the code-review, spec-audit, and TDD verification protocols themselves
And this isn't theoretical! Maintainers have merged or released fixes for 27 bugs that the Quality Playbook found in Javalin, the Linux kernel, Google's gson, aiohttp, calibre, OpenTelemetry and five other projects. Two of them now have CVE IDs: one credited only to the Quality Playbook's report, and one shared with independent reporters.
Install into your project's root directory (pick one):
npx quality-playbook install --into . --ai-tool claude # from npm, no global install
pip install quality-playbook && quality-playbook install --into . --ai-tool claude # from pip
git clone https://github.com/andrewstellman/quality-playbook # or clone and ask your AI to install itSwap claude for cursor, copilot, continue, codex, windsurf, cline, or aider. You need Python 3.10 or later on your PATH.
Run by opening your project in your AI coding tool and telling the agent:
"Run the Quality Playbook on this project."
The agent finds the installed skill on its own and runs the six phases (explore, generate, code review, spec audit, reconcile, verify), pausing after each one until you say "keep going". It writes the results into a quality/ folder. Start with quality/BUGS.md, which lists the bugs it found.
As of 6 November 2026, maintainers have merged or released fixes for these 27 bugs, and 25 more fixes and reports are submitted and waiting. Three of them were private security reports, published by the maintainer as advisories once the fix was released, and GitHub has issued CVEs for the two addressable advisories: CVE-2026-126349 (GHSA-539f, credited only to my report) and CVE-2026-126350 (GHSA-j82w, credited to my report and two independent reporters). The full record, including the bugs that were held back, dropped or never filed, is in evidence/README.md.
| Project | The bug | Fix | Merged | Evidence |
|---|---|---|---|---|
Linux kernel, virtio-pci |
vp_interrupt() returned IRQ_NONE for a config-change interrupt with no vring work, even though reading the ISR had already consumed it, so the kernel counted those interrupts as spurious |
93fa0945 |
2026-09-07 | Evidence folder, with the QEMU console logs |
Linux kernel, nvmet-rdma |
A wrapping offset check let nvmet_rdma_map_sgl_inline() write out of bounds (found independently; an earlier report by Bryam Vargas is credited) |
d1512773, queued for 7.3-stable |
v7.4-rc1 | Evidence folder |
Linux kernel, zram |
recompress_store() silently accepted an unrecognized type= value, leaving the pass with no slot filter so it recompressed every slot instead of the intended subset |
2f529e73 |
2026-04-18 | Evidence folder and the patch thread |
Google gson |
gson silently accepted duplicate keys when the first value was null |
#3006 | 2026-04-05 | Evidence folder |
aio-libs aiohttp |
StreamReader.readuntil() missed a separator of two or more bytes when it was split across chunks, so it returned too much data or kept waiting |
#13870, backported to 3.15 and 3.14 | 2026-10-02 | Evidence folder |
calibre |
The content server returned a 500 instead of a 404 for a malformed hex id in an OPDS feed URL | #3310 | 2026-09-29 | Evidence folder |
| OpenTelemetry Java instrumentation | The Forwarded host extractor recorded "[2001" as the host for a bracketed IPv6 address |
#20393 | 2026-10-16 | Evidence folder |
| OpenTelemetry Java instrumentation | UrlParser recorded server.address as "[" for a bracketed IPv6 address |
#20320 | 2026-10-16 | Evidence folder |
adonisjs http-server |
The uuid route matcher accepted letters that are not hex digits |
#144 | 2026-10-14 | Evidence folder |
python-poetry tomlkit |
Arithmetic on a float written with an explicit sign wrote a negative-zero result as +-0.0, which tomlkit itself then rejected |
#629 | 2026-10-12 | Evidence folder |
harrisiirak cron-parser |
String dates in the RFC 850 and asctime HTTP formats threw "unhandled timestamp", although cron-parser's README lists HTTP dates as supported | #455 | 2026-10-11 | Evidence folder |
bradymholt cRonstrue |
A list with a plain first value and a step printed "At 09:00 AMonly on Friday and , every 2 days of the week" | #423 | 2026-10-20 | Evidence folder |
sporkmonger addressable |
route_from returned an empty string when only the base URI had a query |
#623 | 2026-10-31 | Evidence folder |
sporkmonger addressable |
display_uri showed hosts containing bidi and invisible characters decoded from Punycode, so a host could be made to look like a different one (security advisory, low) |
GHSA-539f (CVE-2026-126349), fixed in 2.9.1 | 2026-10-30 | Evidence folder |
sporkmonger addressable |
Template#match still backtracked exponentially for {/var*}, {.var*} and {;var*} after the 2.9.0 ReDoS fix (security advisory, moderate; two other reporters found overlapping cases) |
GHSA-j82w (CVE-2026-126350), fixed in 2.9.1 | 2026-10-30 | Evidence folder |
harrisiirak cron-parser |
A crontab line or variable named __proto__ replaced the prototype of CronFileParser's errors and variables result objects instead of being stored as an entry (security advisory, low) |
GHSA-vh7f, fixed in 5.10.3 by #461 | 2026-10-31 | Evidence folder |
javalin |
A percent-encoded or double-slash path served a static file without running the path-scoped before-handler that guards it | #2665, released in 7.2.4 | 2026-10-22 | Evidence folder |
javalin |
BasicAuthPlugin let every request through when no username and password were configured |
#2666, released in 7.2.4 | 2026-10-22 | Evidence folder |
javalin |
A HEAD request answered by a GET route was not checked against the GET route's roles | #2668, released in 7.2.4 | 2026-10-22 | Evidence folder |
javalin |
HttpResponseException messages were returned unescaped as text/html |
#2669, released in 7.2.4 | 2026-10-22 | Evidence folder |
javalin |
A carriage return in a server-sent-events comment could start a new event field | #2670, released in 7.2.4 | 2026-10-22 | Evidence folder |
javalin |
Basic auth credentials were ignored unless the scheme was spelled exactly Basic |
#2671, released in 7.2.4 | 2026-10-22 | Evidence folder |
javalin |
The lowercase-path redirect dropped the context path | #2660, released in 7.2.4 | 2026-10-22 | Evidence folder |
javalin |
A static hostedPath was matched as a string prefix, so files were served at sibling URLs without their before-handler |
#2672, released in 7.2.4 | 2026-10-23 | Evidence folder |
javalin |
The precompression cache was shared across static-file configs, so one config's file could be served for another's | #2673, released in 7.2.4 | 2026-10-23 | Evidence folder |
javalin |
Any request with a Sec-WebSocket-Version header created an HTTP session that never expired |
#2674, released in 7.2.4 | 2026-10-23 | Evidence folder |
javalin |
UploadedFile.filename() kept directory components such as ../ from the client |
#2675, released in 7.2.4 | 2026-10-23 | Evidence folder |
Open-source maintainers are drowning in AI-generated pull requests right now, and I have no intention of adding to that pile. The playbook's agents reproduced each of these bugs and drafted the patches, but before anything went upstream I read the code, checked that each regression test failed without the fix and passed with it, and checked the output against the finding. For the virtio bug that meant booting a patched kernel under QEMU. Every one of these fixes says in its pull request or commit message that AI was involved, except the zram patch, which went in before I started requiring that.
Every bug in this table and the next has a record in this repository, which the evidence column links to: the reproduction logs, the patch as sent, and (for all but virtio, zram and gson, which came before I added that step) the review that preceded submission. The gson and zram records are the raw run output, recovered from a backup after the fact; each folder's NOTE.md says what is and is not there.
- Quick start
- Merged bugs leaderboard
- How to install the Quality Playbook
- How to run the Quality Playbook
- Need help? Just ask your AI
- Running the playbook
- What the playbook produces
- How it works
- Want to learn more?
- Submitted upstream
- Recent releases
- Running across many repos with arunner
- Repository structure
- How I improve the playbook
- Context
- License
- Patent notice
The fastest way to install is from npm or pip. From your project's root directory, run one of these:
# From npm, no global install:
npx quality-playbook install --into . --ai-tool <tool>
# From pip / uvx / pipx (Python 3.10+):
uvx quality-playbook install --into . --ai-tool <tool> # one-shot, no global install
pipx run quality-playbook install --into . --ai-tool <tool>
pip install quality-playbook && quality-playbook install --into . --ai-tool <tool><tool> is one of claude, cursor, copilot, continue, codex, windsurf, cline, or aider. The skill installs into .<tool>/skills/quality-playbook/ (or .github/skills/quality-playbook/ for copilot). These commands install it for Claude Code, Cursor and GitHub Copilot:
npx quality-playbook install --into . --ai-tool claude # Claude Code
npx quality-playbook install --into . --ai-tool cursor # Cursor
npx quality-playbook install --into . --ai-tool copilot # GitHub CopilotYou can also install the Quality Playbook (QPB) from the Claude Code plugin marketplace. If you use Claude Code, the first command below adds the marketplace and the second installs QPB from it:
/plugin marketplace add https://github.com/andrewstellman/quality-playbook
/plugin install quality-playbook
Use the full https:// URL, because the shorthand github.com/... form makes Claude Code attempt an SSH clone, which fails on machines without a GitHub SSH key configured.
After you install it this way, the agent finds the QPB skill in any project you open with Claude Code, so you don't need to run quality-playbook install --into .... When a new QPB version ships, run /plugin marketplace update and then install it again.
To work from a local checkout of QPB (for example, to test a fork), use --plugin-dir instead of the marketplace:
claude --plugin-dir /path/to/quality-playbook/plugins/quality-playbook
The --plugin-dir argument takes an absolute path to the plugin directory inside the QPB clone, and it loads the plugin for that Claude Code session only.
You can also ask your AI coding tool to install it from a clone of this repo.
-
Clone this repo somewhere on your machine (for example,
git clone https://github.com/andrewstellman/quality-playbook ~/quality-playbook). You can install into any number of projects from one clone. -
Open your target project in Claude Code, Cursor, GitHub Copilot, Windsurf, Continue, or another AI coding tool.
-
Ask the AI to install it. You can say something like this:
"Install the Quality Playbook into this project from
~/quality-playbook."The agent reads
AGENTS.md, figures out which install location your tool uses, and runs the installer.
If you'd rather install by hand or run the script yourself, load ai_context/TOOLKIT.md into your AI tool (there's more about that file under Need help? below) and ask it: "Read TOOLKIT.md. How do I install the skill manually or via the bin/install_skill.py script directly?"
Prerequisite: You need Python 3.10 or later on your PATH. The npm package is a thin shim over the same Python installer, so npx needs Python 3.10 too.
The playbook finds more bugs when you give it documentation. It reads written specs, design docs, GitHub or Jira issues from real users, chat history, and post-mortems, and from those it works out what your code is supposed to do. Without documentation it still runs from the source tree alone, but it finds fewer bugs. Put everything you have into a reference_docs/ directory at the project root.
One prompt gathers the documentation for you. Copy references/DOC_GATHERING_PROMPT.md into your AI coding tool with your project open, and run it. The agent confirms which project you mean, then collects the project's docs, issues, and advisories into reference_docs/.
Open your project in your AI coding tool (Claude Code, Cursor, GitHub Copilot, Windsurf, Continue, etc.) and tell the agent:
"Run the Quality Playbook on this project."
That one line is enough, because once the skill is installed the agent finds it on its own; you don't have to open or point at SKILL.md or any other file. The agent runs the six phases, pausing after each one until you say "keep going", and writes the results into a quality/ folder in your project.
A full six-phase run takes a while and uses a lot of tokens. To spread it across sessions, for example to stay inside a daily token budget, tell the agent to run only some of the phases:
"Run phases 1 to 3 of the Quality Playbook on this project."
Then, in a later session:
"Continue the Quality Playbook from phase 4."
When the run finishes, the quality/ folder contains:
quality/
βββ BUGS.md β consolidated bug report with spec basis (start here)
βββ REQUIREMENTS.md β behavioral requirements derived from your code + docs
βββ EXPLORATION.md β Phase 1 findings: patterns explored, files tagged
βββ QUALITY.md β quality constitution for your codebase
βββ CONTRACTS.md β extracted behavioral contracts
βββ COVERAGE_MATRIX.md β contract-to-requirement traceability
βββ COMPLETENESS_REPORT.md β final gate report with post-reconciliation verdict
βββ PROGRESS.md β phase checkpoint log + cumulative bug tracker
βββ test_functional.py β functional tests traced to requirements
βββ test_regression.py β regression tests for confirmed bugs
βββ writeups/ β per-bug detailed writeups with patches (BUG-NNN.md)
βββ patches/ β fix and regression-test patches
βββ code_reviews/ β three-pass code review output
βββ spec_audits/ β Council of Three auditor reports + triage
βββ results/ β TDD red/green logs, integration results, gate log
Start with BUGS.md for the headline findings. Then read REQUIREMENTS.md, the playbook's account of what your code is supposed to do, which includes requirements it derived from issues and docs you may not have known about. The bugs it reports come from the gap between what REQUIREMENTS.md says and what your code does.
The rest of this README covers the playbook's phases and output files, but the easiest way to get answers is to download one file, attach it to your favorite AI chatbot, and ask it whatever you want to know.
The file is ai_context/TOOLKIT.md. It's a single Markdown document that explains the Quality Playbook, written so that an AI assistant can read it and answer your questions about the playbook.
Open a chat in whatever AI tool you use, attach TOOLKIT.md, and tell it:
"Read TOOLKIT.md. Now you're an expert in the Quality Playbook."
Then ask it anything, from how to set the playbook up to why your run found only one bug. The Want to learn more? section below has example prompts.
This ChatGPT conversation shows what that looks like, and it works the same way in other AI tools.
The playbook runs in six phases, each in its own context window. After each phase, say "keep going" to continue.
| Phase | Purpose |
|---|---|
| 1. Explore | The agent reads the code, tests, config, specs, commit history, and reference_docs/ if it's there, to work out both what the code does and what it's supposed to do. |
| 2. Generate | A five-stage pipeline derives the behavioral requirements, then generates the functional and integration tests, the code-review, spec-audit and TDD protocols, and the quality constitution. |
| 3. Code review | Three-pass review: structural, requirement verification, cross-requirement consistency. Each confirmed bug gets a regression test. |
| 4. Spec audit | Three independent AI models audit the code against the requirements (the Council of Three). Triage uses verification probes ("is this actually true?"), and the models don't vote. |
| 5. Reconciliation | Every bug from the code review and the spec audit is tracked, and either gets a regression test or is explicitly exempted. |
| 6. Verify | 45 self-check benchmarks validate generated artifacts. Final gate: GATE PASSED / GATE PASSED WITH CLEANUP NEEDED / GATE PASSED WITH DECISIONS NEEDED / GATE FAILED. |
After Phase 6 the agent presents a summary of the results and a menu of next steps (SKILL.md calls this Phase 7). Since v1.6.1 it can also prepare an upstream submission for one confirmed bug at a time: a policy check, confirmation, a fix, a blind review panel and a draft PR. It never submits anything itself.
After the first six-phase run (the baseline), you can run four iteration strategies in order: gap, unfiltered, parity, and adversarial. Each strategy looks for different classes of bug, and running all four typically adds 40 to 60 percent more confirmed bugs on top of the baseline. Say "Run the next iteration using the gap strategy" to begin.
After you fix bugs, say "recheck". Recheck mode checks your fixes against the existing bug report without re-running the full pipeline, and takes 2 to 10 minutes.
To run the playbook autonomously or in CI, use the prompt under Running autonomously or in CI in Want to learn more? below, or the orchestrator agent in plugins/quality-playbook/skills/quality-playbook/agents/quality-playbook.agent.md.
A full baseline run takes hours and uses a large share of a daily or weekly quota on most providers. One measured example: the run on Javalin (about 13,000 lines of non-test Kotlin and Java) on 29 and 30 September 2026, with QPB 1.6.0 on Claude Opus 5.5, took about 24 hours from start to finish and used the equivalent of about $94 at API prices. That run found the Javalin bugs in the leaderboard above.
For multi-repo benchmark work, see harness_plans/ for parallel-pool plan formats and docs/design/QPB_Test_Harness_*.md for the harness internals.
The phase checks rely on the agent following written instructions; the code doesn't enforce them. Phases 1, 2, and 5 tell the agent to run validate_phase_artifacts and quote its verdict line, and Phase 6 tells it to use a fresh-context auditor sub-agent. In one Phase 1 run, Codex desktop reported PASS without running the validator. When you review a phase verdict, check for the verbatim RESULT: VALIDATION PASSED (phase N) line, or for fresh-context framing in the Phase 6 auditor verdict.
The proposal for enforcing these checks in code is in docs/design/QPB_v1.6.x_Phase6_Structural_Enforcement_Proposal.md.
The playbook generates these files:
| Artifact | Location | What it does |
|---|---|---|
REQUIREMENTS.md |
quality/ |
Behavioral requirements derived from code, docs, and community sources via a five-stage pipeline. Without requirements, review can find only structural bugs. |
QUALITY.md |
quality/ |
Quality constitution defining what "correct" means for this specific project, with fitness-to-purpose scenarios and rules against coverage theater (tests that look thorough but aren't). |
test_functional.* |
quality/ |
Functional tests in the project's native language, traced to requirements rather than generated from source code. |
RUN_CODE_REVIEW.md |
quality/ |
Three-pass protocol: structural review, requirement verification, cross-requirement consistency. Each pass is designed to find bugs the other two miss. |
RUN_SPEC_AUDIT.md |
quality/ |
Council of Three: three independent AI models audit the code against requirements. Different models have different blind spots, and the triage uses verification probes (targeted checks that ask "is this actually true?"), and the models don't vote. |
RUN_INTEGRATION_TESTS.md |
quality/ |
End-to-end test protocol grounded in use cases, with a traceability column mapping each test to the user outcome it validates. |
RUN_TDD_TESTS.md |
quality/ |
Red-green TDD verification protocol that, for each confirmed bug, shows the regression test failing on unpatched code and passing with the fix. |
BUGS.md |
quality/ |
Consolidated bug report with spec basis, severity, reproduction steps, and patch references for every confirmed finding. |
AGENTS.md |
project root | Bootstrap file so every future AI session inherits the full quality infrastructure. |
The playbook's value comes from requirement derivation. AI code reviewers have the same limit human reviewers do. If you don't know what the code is supposed to do, you can only find structural issues. The playbook's main job is working out that intent from your code and your documentation, and then building every later artifact from it.
For more detail on any of the phases in the Running the playbook table above, ask your AI tool:
"Read TOOLKIT.md. Walk me through Phase N in detail. What's the input, what's the output, what can go wrong?"
The final gate produces one of four verdicts:
GATE PASSED: the review is complete and there is nothing to do.GATE PASSED WITH CLEANUP NEEDED: the bug findings are real and stand on their own, and only the audit trail is incomplete.GATE PASSED WITH DECISIONS NEEDED: there is no substantive problem, but some requirements claim more than the passage they cite. No confirmed bug rests on those requirements, andquality/OPERATOR_DECISIONS.mdlists your options for each one.GATE FAILED: a substantive problem, such as an incomplete review, missing specs, or a fabricated verdict.
The four verdicts separate "your code is broken in N ways" from "your audit trail is incomplete in N ways". Every verdict starts with a ββ Bug evidence ββ block that says how many bugs have red and green test logs; that count is reported on its own, apart from the pass or fail.
In a controlled experiment across multiple repositories, runs with documentation found more bugs than the code-only runs, and different ones, and reported them with higher confidence. The documentation gives the auditors spec language to check against, so "this code looks odd" becomes "this code contradicts the documented behavior".
You can answer most questions by loading TOOLKIT.md into your AI tool and asking. These example prompts cover common situations:
Setting up documentation:
"Read TOOLKIT.md. What documentation should I provide and how do I gather it? What's the difference between top-level
reference_docs/andreference_docs/cite/? What about projects with no documentation?"
Understanding what each phase does:
"Read TOOLKIT.md. What does Phase 3 actually do? How does it find bugs that structural code review misses? How is it different from Phase 4?"
Choosing iteration strategies:
"Read TOOLKIT.md. What are the four iteration strategies (gap, unfiltered, parity, adversarial)? When should I use each? What's the right order?"
Running autonomously or in CI:
"Read TOOLKIT.md. How do I run the playbook autonomously across multiple iterations without babysitting it? What about CI?"
Interpreting low bug counts:
"Read TOOLKIT.md. My run only found one bug. Is my code that clean, or did the playbook miss something? How do I tell?"
The Council of Three:
"Read TOOLKIT.md. What is the Council of Three? Why three models? How does the triage handle disagreements?"
Manual install fallback:
"Read TOOLKIT.md. The automatic install didn't work. How do I copy the skill files manually for Claude Code, Cursor, GitHub Copilot, Continue, Windsurf, Codex, Cline, or Aider?"
Tuning recall:
"Read TOOLKIT.md. My playbook missed a bug I know is in the code. What can I tune on the next run?"
Understanding the gate verdicts:
"Read TOOLKIT.md. What's the difference between GATE PASSED, GATE PASSED WITH CLEANUP NEEDED, GATE PASSED WITH DECISIONS NEEDED, and GATE FAILED?"
If TOOLKIT.md doesn't answer your question, file an issue at https://github.com/andrewstellman/quality-playbook/issues.
These fixes and reports are still waiting on the maintainers. All of them except those opened in November (setuptools #5374 and #5375, zod #6767, #6769 and #6770, tomlkit #714, addressable #702 and #703) were checked on GitHub on 27 October 2026, and none had been merged or closed. The Linux patches were checked against the history of mainline (torvalds/linux), not their mailing-list threads. The five marshmallow pull requests from #3086 to #3090 share one evidence folder: three of them were first reported privately to marshmallow's security contact, who ruled them not security issues and asked for public pull requests. Reports that may be security issues and are still private are not listed until the maintainers publish them.
| Project | The bug | Submitted | Evidence |
|---|---|---|---|
wanasit chrono |
With a reference timezone that differs from the system's, "this month", "last night" and dates without a year used the system timezone's calendar | #670 | Evidence folder |
RubyMoney money |
Dividing a subclass of Money by a plain Money returned a Money instead of a Float |
#1237 (approved by a maintainer, with a suggested extra test) | Evidence folder |
javalin |
enableBasicAuth did not protect WebSocket upgrades |
#2667 (held by the maintainer for Javalin 8, where it changes documented behaviour) | Evidence folder |
marshmallow |
URL(relative=True) rejected a relative URL with "://" in its path, query or fragment |
#3067 | Evidence folder |
marshmallow |
A pre_load(pass_collection=True) hook that changes the data's length made a per-item @validates_schema raise ValueError under many=True |
#3086 | Evidence folder |
marshmallow |
A @validates method never ran for a field with a dotted attribute |
#3087 | Evidence folder |
marshmallow |
A value rejected by @validates on a field with attribute set stayed in ValidationError.valid_data |
#3088 | Evidence folder |
marshmallow |
Nested dropped dotted load_only and dump_only for a nested schema instance |
#3089 | Evidence folder |
marshmallow |
Under unknown=INCLUDE, an unknown key that matched a loaded field's key overwrote the loaded value |
#3090 | Evidence folder |
pypa setuptools |
readme or requires-python set outside dynamic crashed instead of being warned about and ignored |
#5343 (the maintainer has pushed review changes to it) | Evidence folder |
pypa setuptools |
MANIFEST.in exclude patterns ignored character ranges such as [a-c] |
#5374 (an outside reviewer caught a reversed-range edge case, fixed in a follow-up commit) | Evidence folder |
pypa setuptools |
exclude_package_data didn't exclude a file whose name is stored in NFD, as macOS does, when the pattern is NFC (first reported privately to setuptools' Tidelift contact, who said to make it public) |
#5375 | Evidence folder |
spf13 cobra |
Command suggestions measured edit distance in bytes, not characters | #2514 | Evidence folder |
colinhacks zod |
registry.remove() deleted an id that now belonged to another schema |
issue #6640 (zod accepts pull requests only from collaborators) | Evidence folder |
colinhacks zod |
An intersection threw when both sides produced NaN |
issue #6639 | Evidence folder |
colinhacks zod |
Compiled parsing accepted a missing key whose optional was inside z.intersection() or z.success() |
issue #6767 (fix on a branch of my fork) | Evidence folder |
colinhacks zod |
An exhaustive z.record() accepted a missing key when the value schema accepted undefined |
issue #6769 (fix on a branch of my fork) | Evidence folder |
colinhacks zod |
A constant .catch() value and a Date default were shared across parses |
issue #6770 (fix on a branch of my fork) | Evidence folder |
python-poetry tomlkit |
parse() raised RecursionError on a valid document whose dotted key was within the nesting limit |
issue #714 | Evidence folder |
sporkmonger addressable |
normalize decoded %3A, %5B and %5D in a host name, so an encoded colon became a port |
#702 | Evidence folder |
sporkmonger addressable |
The pure-Ruby to_ascii skipped the characters nameprep removes, such as a zero width joiner |
#703 | Evidence folder |
NVIDIA bionemo-recipes |
The THD context-parallel split dropped the remainder tokens | #1765 | Evidence folder |
NVIDIA bionemo-recipes |
AMPLIFY _pad_weights returned fp32 for bf16 input |
#1764 | Evidence folder |
Linux kernel nvmet |
ANA group ID 128 was clamped to 0 by array_index_nospec() |
v2 applied to nvme-7.4 by Keith Busch, 6 November 2026, not yet in mainline | Evidence folder |
Linux kernel nvmet |
The CRTO property was computed from CSTS instead of CAP | patch sent to linux-nvme, 10 September 2026 | Evidence folder |
These are the highlights; CHANGELOG.md has the detail for every release.
- v1.6.1: the release built from the open-source bug campaign. Sub-agents under the parent-witness rule: phases may run in per-phase sub-agents, as long as the parent session that spawned them re-runs the gate itself after Phase 6 (
scripts/qpb_gate_witness.pyin the plugin's skill folder). Preparing upstream submissions: a new path after the six phases takes one confirmed bug at a time through a policy check, confirmation, a fix, a blind review panel and a prepared draft PR. The skill never submits anything upstream itself; five of its guardrails are written intoSKILL.mdand the rest are inreferences/upstream_submission.md. The release also adds rules for the claims QPB makes in its own output, a gate verdict that reports decisions and bug evidence plainly, and fixes to the persona validation that v1.6.0 introduced. - v1.6.0: the requirements review and grounding release. Dump-and-go documentation classification: drop any docs in
reference_docs/and the model reads and classifies each one. The model cites authoritative references (markedunconfirmeduntil you confirm them), leaves CVEs, advisories and source code for you to decide on, and doesn't demote a real reference just because the project's maintainers didn't write it. Agent-persona requirements validation: fresh-context domain-expert and security personas independently check the derived requirements and correct them, with every change grounded in a document and revertable. The release also adds a mechanically checked structure for the requirements document, an opt-in interview in which you validate the requirements, and durable evidence of what you confirmed. - v1.5.10: repo hygiene. The canonical
SKILL.mdandreferences/moved back to the repo root as the single source of truth (the in-tree skill locations are symlinks),SKILL.mdwas trimmed, with per-phase detail loaded fromreferences/*.mdas needed, and run output and orphaned partial copies that had been committed were removed from tracking. - v1.5.8: distribution channels. QPB was published to pip (PyPI), npm, and the Claude Code plugin marketplace, with publish scripts that require an explicit
--publishafter a--dry-run(and--otpfor 2FA), and the repository was restructured to Claude Code's standard self-hosted plugin marketplace layout (plugins/quality-playbook/). - v1.5.7: channel scaffolding for pip, uvx, pipx and npx, phase-aware bundling, the Council of Three review protocol was formalized, and the worker self-Council pattern (parallel sub-agent reviewers through the
Tasktool) was added. - v1.5.6: the improvement loop formalized. The v1.5.x releases build the quality-control infrastructure (finding bugs, validating the skill's prose), and v1.6 onward builds quality improvement on top of it (statistical control, multi-operator workflows).
For multi-repo runs, benchmark suites, or unattended batch work, drive the playbook with arunner, a batch orchestrator for agentic coding systems, written with only the Python standard library. You point it at a list of jobs (run the playbook across these ten repos, audit these branches), and it runs them in a pool, watches each job's heartbeat, and leaves a complete record on disk. It needs no server, no daemon, and no API key beyond your existing agent session, and to recover from a crash you run one tick (one pass of its scheduler, which is idempotent): if you close the window or the machine goes to sleep, arunner picks up where it left off.
arunner is vendor-neutral because its orchestration engine is plain Python and a job is anything that appends JSON lines to a file, so a QPB run, a shell script, or a CI job all qualify. I recommend it for running QPB across a set of repos without babysitting each one.
See the arunner repository for the worker contract, plan formats, and host support table. Native support for running QPB's phases through arunner is still in development. Until it ships, run QPB through arunner as a shell or agent job, or run it phase by phase as described in How to run the Quality Playbook above.
The canonical SKILL.md and its references/ directory are at the repo root, and the skill folders inside the plugin tree are symlinks back to them. QPB uses Claude Code's standard self-hosted marketplace layout: .claude-plugin/marketplace.json points at the plugin under plugins/quality-playbook/. The pip and npm packages copy the real files from the root, with the symlinks resolved.
quality-playbook/
βββ SKILL.md # The skill: canonical source of truth (full operational instructions)
βββ references/ # Protocol + per-phase pipeline reference docs (lazy-loaded per phase)
β βββ DOC_GATHERING_PROMPT.md
β βββ exploration_patterns.md
β βββ requirements_pipeline.md
β βββ review_protocols.md
β βββ spec_audit.md
β βββ verification.md
β βββ phase1_exploration_guide.md
β βββ phase2_generation_guide.md
β βββ phase5_reconciliation_guide.md
β βββ phase6_verify_guide.md
β βββ phase7_guide.md
β βββ recheck_mode.md
β βββ run_state_schema.md
β βββ ... (other reference docs)
βββ schemas.md # Artifact schema definitions
βββ AGENTS.md # AI bootstrap file (repo root)
βββ .claude-plugin/ # ROOT-only: marketplace catalog
β βββ marketplace.json # Self-hosted marketplace entry (source: ./plugins/quality-playbook)
βββ plugins/ # Plugin tree (standard self-hosted marketplace layout)
β βββ quality-playbook/ # The plugin itself
β βββ .claude-plugin/
β β βββ plugin.json # Plugin metadata (name, description, version, author)
β βββ skills/
β βββ quality-playbook/
β βββ SKILL.md # Symlink β repo-root SKILL.md
β βββ references/ # Symlink β repo-root references/
β βββ phase_prompts/ # Per-phase agent prompts (Mode A + Mode B)
β βββ agents/ # Orchestrator agent files for autonomous runs
β βββ ai_context/ # Adopter-facing AI context (TOOLKIT.md symlink)
β βββ scripts/ # Bundled scripts (quality_gate.py, install_skill.py, ...)
βββ bin/ # Repo-level runner + build scripts (Python 3.10+)
β βββ run_playbook.py # Mode B runner (positional args are target directories)
β βββ build_channel_package.py # Stages the pip/npm bundle (dereferences symlinks)
β βββ publish_pip.py # Pip publish path
β βββ publish_npm.py # Npm publish path
β βββ install_skill.py # Thin shim that delegates to the plugin's scripts/install_skill.py
β βββ tests/ # stdlib-only unit tests (python3 -m pytest bin/tests/)
βββ pytest/ # Local stdlib-only shim (python3 -m pytest works without installs)
βββ ai_context/ # Maintainer orientation docs + adopter-facing TOOLKIT.md
β βββ DEVELOPMENT_CONTEXT.md
β βββ DEVELOPMENT_PROCESS.md
β βββ IMPROVEMENT_LOOP.md
β βββ TOOLKIT_TEST_PROTOCOL.md
β βββ BENCHMARK_PROTOCOL.md
β βββ TOOLKIT.md # Adopter-facing AI context; symlinked into the skill bundle
βββ harness_plans/ # Parallel-pool benchmark plan formats
βββ runner/ # arunner run directories (per-version harness state)
βββ docs/ # Design docs, implementation plans, proposals
βββ reviews/ # Council-of-Three review artifacts
βββ metrics/ # Cross-version benchmark metrics
βββ images/ # README images
βββ reference_docs/ # Documentation QPB reads to derive requirements
βββ scripts/ # Repo-level helper scripts
βββ quality_playbook_cli/ # CLI package
βββ quality/ # quality/audits/ only; the self-run output was removed from tracking in v1.5.10
βββ pyproject.toml # Pip packaging + bundle globs
βββ package.json # npm shim packaging
βββ CHANGELOG.md # Per-release detail
βββ LICENSE.txt # Apache 2.0
Each release goes through a Plan-Do-Check-Act loop, and the Check step is benchmark recovery against pinned ground truth. I make a change and run the playbook against three pinned benchmark repositories (chi-1.5.1, virtio-1.5.1, express-1.5.1) whose bug counts are already known from earlier releases. The release ships only if both verification dimensions, process compliance and outcome recall, hold or improve.
Verification dimensions are what I measure on every release: process compliance (does the run produce the right artifacts?) and outcome recall (does the run find the bugs I know are there?). The most dangerous failure is a run that passes on process but fails on recall, which means every gate is green while the run found no real bugs.
Improvement levers are what I change to make the playbook better. Each lever lives in a known place in the codebase and can be tuned without affecting the others. The current levers are:
- exploration breadth and depth (
references/exploration_patterns.md,references/iteration.md) - code-derived versus domain-derived requirements (
references/requirements_*.mdplus the skill'sscripts/citation_verifier.py) - gate strictness (the skill's
scripts/quality_gate.py) - finalization robustness (
bin/run_playbook.py::_finalize_iteration) - the mechanical citation extractor (
bin/skill_derivation/citation_search.py, with its token-overlap pre-filter) - the four-pass skill-derivation pipeline (
bin/skill_derivation/pass_{a,b,c,d}.pyplus the divergence-detection modules underbin/skill_derivation/divergence_*.py)
I connect the levers to outcome recall with regression replay. I take a pinned benchmark, roll it back to a commit just before a known bug was fixed, and run the playbook against that pre-fix commit. If the playbook finds the bug, the levers are good enough for that class of bug. If it misses the bug, I work out which lever to change, change it, and re-run the playbook to check both that it now finds the bug and that recall on the rest of the benchmark holds.
ai_context/IMPROVEMENT_LOOP.md has the full detail, including the lever inventory with file mappings and the plan for moving to formal statistical process control. The review that checks the ai_context/ orientation docs before each release (the documentation version of the Council of Three) is described in ai_context/TOOLKIT_TEST_PROTOCOL.md.
This project accompanies the O'Reilly Radar article AI Is Writing Our Code Faster Than We Can Verify It, part of my series on AI-driven development for O'Reilly Radar. I built the playbook using AI-driven development, with Octobatch, an open-source Python batch LLM orchestrator. This README was coauthored with Claude.
Apache 2.0.
Aspects of the methodology described in this repository are the subject of US Provisional Patent Application No. 64/044,178, filed April 20, 2026 by Andrew Stellman.
Users of this project are covered by the Apache License 2.0, which includes an express patent grant in Section 3. That grant is perpetual, worldwide, royalty-free, and irrevocable (except as described in the license), and extends to anyone using, reproducing, modifying, or distributing the Quality Playbook under the terms of the Apache 2.0 license. Nothing in this notice diminishes that grant.
The patent application exists to preserve a defensive priority date; it is not asserted against users, contributors, forks, or derivative works of this project practiced under Apache 2.0.
