Mutation testing insights, release notes, and findings from the moonbuggy project.
Last month we wrote that the speed race in Python mutation testing runs on honor-system math: self-reported numbers, no shared harness, every claim a claim until someone reproduces it. The honest response to that post is to hold your own numbers to the same standard, so we do. There’s a nightly CI job whose entire job is to check that our speed still holds, and it re-primes a baseline from real runs instead of trusting the README.
277.8 seconds. That’s what it took moonbuggy to generate 381 mutants of more-itertools and run the library’s own test file against every one of them. The same job took a naive re-run-everything baseline 3,716.4 seconds. mutmut, in between, took 333.6 seconds, but it also generated 1,085 mutants, because its operator set is bigger. Those two numbers are not the same job, and this post is not going to pretend they are.
The most useful thing our mutation tool found in five real Python libraries was three bugs in itself.
That’s not false modesty and it’s not a punchline. It’s the honest read of the numbers, and the numbers are worth walking through, because they say something about what mutation testing is actually for.
We ran moonbuggy against five real open-source libraries: tomli, humanize, sqlparse, more-itertools, and boltons. Each pinned at a tagged release, cloned read-only, in an isolated venv, with the project’s own test suite green before any mutant ran. 1,313 mutants total. Scores ranged from 0.64 (boltons) to 0.91 (tomli).
pytest-gremlins reports 13.82x speedup with caching and 3.73x parallel versus mutmut. fest, a Rust-powered mutation tester for Python, launched on Reddit claiming roughly 25x faster than cosmic-ray. Both are interesting tools. Neither number has been reproduced by anyone outside the project that published it. That includes us.
We’re not picking on either project. The whole speed race in Python mutation testing runs on honor-system math. Every tool benchmarks itself, on its own suite, against whatever baseline it picks, and publishes the winner. There is no shared harness, no third party running the same workload through gremlins, fest, mutmut, cosmic-ray, and anyone else with a claim. Until someone does, every number in this market is a claim, and that goes for the ones on our README too.
“KILLED” is the weakest sentence in mutation testing, and it’s the only one most tools say.
Think about what it actually tells you. A mutant died. That’s the whole message. Not which test killed it, not which assertion did the damage, not whether the death was meaningful or noise. If your codebase has a survivor, the tool shrugs and hands you a file name, and you go read the code yourself to find out whether the mutant deserved to live.
The moonbuggy blog is where we share what we learn while hunting surviving mutants. Expect:
The documentation tells you how moonbuggy works. The blog tells you what we’re finding with it. A SURVIVED line in the output names a gap in your suite — down to the line and operator. The blog is where we tell the story of how we found it, what it means, and how to fix it.