What the source did
- He writes Ceph code for IBM and Red Hat, on erasure coding, including pools that span two data centres. The bugs that matter there show up only when something fails at the wrong moment, like an OSD dying mid-write. Ceph's own integration tests take hours per run, and getting from a failed job to an understood bug can take a developer days.
- From 29 Sep to 12 Oct 2026 he let Claude Code run the whole loop on one machine while he steered: inject faults, notice a break, work out why, turn it into a unit test that fails, fix it, rebuild, repeat.
- The fault injector, vstart_chaos, is a set of scripts Claude Code wrote with him. Claude kept changing it between runs: the first campaign found 16 bugs in the tool itself, such as false "stuck" reports and clients that leaked for hours.
- A cluster failure can't be replayed, so Claude reproduced each one by reasoning. It read the logs and shard dumps, worked out the mechanism and wrote it as a unit test in Ceph's peering and EC harnesses. It didn't move on to a fix until the test failed on main. In his worked example the first test passed, and only a second hit showed the missing condition.
- For theories about where data could go wrong, it added tripwire assertions that crash the OSD at that exact point. These were marked do-not-merge and never left his machine.
- The session running faults acted as a lead and handed diagnosis to read-only subagents. A workflow took one bug at a time: a fixer in its own worktree wrote the failing test and the fix, two reviewers checked it, and a rework stage fixed what they blocked. Helpers never push, file or comment.
- To run unattended: an overnight driver script, a check-in every 30 minutes, evidence captured before the next run wipes the cluster, and a runbook plus one-fact memory notes so a fresh session starts with the rules. Several Claude sessions shared the box, so they booked the one test cluster, kept a disk floor and messaged each other about space.
- He stayed in charge of what to test, what counts as a bug, design choices and everything that leaves the machine. Claude shows him the exact tracker or PR text and he approves each one. PRs start as drafts, and he adds the sign-off himself.
- Results: 55 real defects (50 from the first two sessions, 27 of those in upstream code, plus 5 more on main later). About $840 at API list prices for the first two sessions. His estimate is that the human part fell from about two days per bug to a couple of hours, roughly 100 person-days down to about 13.
- What went wrong: a failure sat unnoticed for six hours because a check ran detached, the disk filled more than once, Claude killed its own shell with a pattern match, and parallel test suites killed each other's processes.
“a yes or no on everything that leaves the machine”— Alex Ainscow (Ceph.io blog), Ceph.io blog
Summary written by Agent Cookbook from the original post. See the original for the full context. How we source recipes
Our notes
- What it takes
- Claude Code with subagents and workflows, a fault injector (his vstart_chaos is linked), a unit-test harness that recreates failures without a live cluster, and a runbook plus memory notes.
- Where it breaks
- Check-ins that wake nobody, disks filling with logs, killing processes by pattern, fixing before a test fails, and write-ups reviewers can't follow. Every fix still needs a person to understand it.
- Human checkpoint
- Claude Code shows the exact text of every tracker, PR and comment and waits. You approve each one, PRs start as drafts, you add the sign-off, and design choices wait for you.
- Tools involved
- Claude Code with subagents and workflows, A fault injector for your system (his is vstart_chaos, linked), A unit-test harness that can recreate a failure without a live cluster, A runbook and memory notes for the rules, A person who approves everything that leaves the machine
✓ Verified Source link checked and still live. Notes by Thomas, editor. How we verify
Get their template
Made and published by Alex Ainscow. Start here; it is their original, not ours.
Repo · by Alex Ainscowvstart_chaos, his fault injector (Ceph PR 72412)Draft PR by Alex Ainscow: src/script/vstart_chaos plus a guide in doc/dev/vstart_chaos.rst. It builds a one-machine Ceph cluster, runs client workloads that check every read, and injects faults in cycles. He notes you don't need an AI to run it.Open it ↗Repo · by Alex AinscowThe worked example: tests-only draft PR 72582The recreate test for the lost-omap-keys bug he walks through in the post, filed with tracker 81630 after he approved it.Open ↗
Our adapted version
Ours Our version, built from his post. He didn't publish his runbook or memory notes; his fault injector is linked above. Fill in the [BRACKETS].
Run a test-and-fix loop on this project. I set the direction; you run the loop. Nothing leaves this machine without my yes.
THE LOOP
1. Run the fault injector: [command]. Run it in the background and check in every 30 minutes, or as soon as the run ends.
2. When a run fails, capture the evidence first (client log, state dumps, each daemon's log for the minutes around the failure) before anything is wiped.
3. Work out the mechanism from the evidence. Hand deep diagnosis to a read-only subagent: it must not edit, build or touch the live system.
4. Write a unit test that recreates the failure without the live system. If it passes on main, the model of the bug is incomplete: extend the harness or wait for another hit. No fix until the test fails.
5. Fix it in its own worktree, one commit per bug, with a control case that passes. Have a separate reviewer check correctness and regressions.
6. If the failure was the tool's fault, fix the tool instead and say so.
RULES
- Confirm a bug empirically before writing a fix.
- Expected behaviours I've ruled on are in [runbook]. Add every new ruling as a one-line memory note.
- When a fix needs a design choice, write up the options and stop.
- Kill processes by exact PID, never by pattern. Stop before the disk drops below [floor].
WHAT NEEDS MY YES
- Filing any tracker, opening any PR (drafts only) or posting any comment: show me the exact title and text and wait.
- Deleting anything you didn't create.
Each morning, list: bugs found, tests written, fixes ready for review, tool fixes, and anything waiting on me.
Approval rule
🛡
Claude Code runs the loop and drafts every tracker, PR and message, then shows you the exact text. You approve each one, PRs start as drafts, you add the sign-off, and design choices wait for you. Sending, buying, posting and deleting always need a human.
Set it up
- Get a fault injector that runs on one machine and checks every read. His is linked above; you don't need an AI to run it.
- Write the rules into a runbook: how to run, how to capture evidence, how to file, and what counts as a bug.
- Start with one supervised run and rule on each failure yourself. Turn each ruling into a memory note.
- Add the overnight driver and the 30-minute check-in, then let it run while you sleep.
- In the morning, review each failing test, fix and draft text, and approve what gets filed.
- Budget time for review. Every fix still needs a person who understands it.
Common failures
⚠
- A check-in that wakes nobody. His once ran detached and a failure sat unnoticed for six hours.
- Filling the disk with logs or parallel test suites. He added log-level limits and a stop below 10 GB free.
- Killing processes by pattern. Claude killed its own shell that way, so it now kills exact PIDs.
- Fixing before a test fails. A recreate test that passes on main means the bug isn't understood yet.
- Write-ups reviewers can't follow. Many of his commit messages and PR descriptions took several tries.
- Treating the bug count as the cost. The bigger cost was the human time to review every fix.
Variations
- Run the fault injector without an AI and hand only the failures to Claude Code.
- Point the same loop at another distributed system or a service with its own fault injector.
- Use tripwire assertions, kept local, to test a theory about where data goes wrong.
Get the 10 best agent setups. Free.
Plus new recipes every week. One email, no spam.
Unsubscribe anytime. Privacy