Quickstart

Audience: you have a Python project with a pytest suite. You have never used mutation testing and do not need to understand it yet.

Goal: get moonbuggy running, read one survivor, and fix it. Ten minutes.

1. Start it

There are two ways in. The first is the one moonbuggy is designed around.

If you are handing this to an agent

Show it the help screen and let it work:

$ uv run --with moonbuggy moonbuggy -h

uv run --with pulls moonbuggy into a throwaway environment for that one command, so there is nothing to install and nothing added to your project. That help screen is the whole interface: no configuration file, no scaffolding step, nothing to read first.

The first real run, from your project root and inside the virtualenv where your tests already run, is a single line:

$ uv run --with moonbuggy moonbuggy --include src/yourpkg --pytest-arg=-q

--include keeps the first run small, and --pytest-arg passes anything your suite needs through to every pytest run. Both are optional.

If you are running it yourself

Install it:

$ pip install moonbuggy

moonbuggy needs Python 3.12 or later and a project whose tests run under pytest. It has no configuration file and does not want one.

2. Run it

From your project root — the directory where you would run pytest:

$ moonbuggy

No flags. moonbuggy finds your source directory, finds your tests, runs them once to see which tests execute which lines, and then starts breaking things.

The first thing it does is check your suite is green. If it is not, it stops and says so, because mutation results against a failing suite are meaningless — see Troubleshooting.

3. Read the output

Every line is one mutation, and the first word is the verdict:

KILLED    app/pricing.py:15 constant_int line=15 nearest_test=- tests_run=3 id=app/pricing.py:15:constant_int:0
SURVIVED  app/pricing.py:14 comparison_swap line=14 nearest_test=tests/test_pricing.py::test_discount tests_run=3 id=app/pricing.py:14:comparison_swap:0

KILLED is good news. moonbuggy changed line 15, a test failed, your suite caught it. Nothing to do.

SURVIVED is the finding. moonbuggy changed line 14, ran the three tests that execute line 14, and all three still passed. Your suite does not notice that change.

NO_COVERAGE is the third verdict you will meet. It means no test executes that line at all, so nothing could have objected. Same exit code as a survivor, different fix: write a test rather than strengthen one.

Only the findings are worth your attention:

$ grep -E '^(SURVIVED|NO_COVERAGE)' .moonbuggy/results.txt

4. Look at one survivor

The line tells you where, but not what changed. Ask:

$ moonbuggy show app/pricing.py:14:comparison_swap:0
id           app/pricing.py:14:comparison_swap:0
status       SURVIVED
location     app/pricing.py:14
operator     comparison_swap
nearest_test tests/test_pricing.py::test_discount
tests_run    3
diff
  - if quantity > 10:
  + if quantity >= 10:

Read that as a question about your tests: if the threshold were >= instead of >, would any test fail? moonbuggy has already answered — no.

So nothing in your suite pins down what happens at exactly 10 items. If the boundary is wrong, or if someone changes it next year, the tests stay green.

5. Fix it

Write the test that would have caught it. The boundary case is the missing one:

def test_exactly_ten_does_not_qualify():
    assert discount(quantity=10) == 0

Now ask whether that test actually kills the mutant, without re-running the other seventy:

$ moonbuggy run app/pricing.py:14:comparison_swap:0
id           app/pricing.py:14:comparison_swap:0
status       KILLED
location     app/pricing.py:14
operator     comparison_swap
tests_run    3
selected     tests/test_pricing.py::test_discount
             tests/test_pricing.py::test_exactly_ten_does_not_qualify
             tests/test_pricing.py::test_bulk
failed       tests/test_pricing.py::test_exactly_ten_does_not_qualify
diff
  - if quantity > 10:
  + if quantity >= 10:

run is show with the run attached: same coverage pass, same selection, same runner, one mutant. It never serves a cached verdict for its target — the point of the command is to re-measure — and it leaves results.jsonl alone, since that file is the record of a run and no run has happened. Exit code is 0 when the mutant is killed and 1 when it is not, so it works in a script.

To re-check everything still outstanding after a round of new tests, pipe the findings in:

$ grep -E '^(SURVIVED|NO_COVERAGE)' .moonbuggy/results.txt | moonbuggy run -

If a survivor refuses to die, do not reach for an experiment — ask:

$ moonbuggy why app/pricing.py:14:comparison_swap:0

why runs nothing. It prints which tests selection picks for that mutant (if your new test is not in the list, that is your answer) and whether the cache already holds a verdict for those inputs (if it does, that is the other one). See Reading the output.

Run moonbuggy again:

$ moonbuggy
moonbuggy: KILLED=72  KILLED_BY_ERROR=0  NO_COVERAGE=0  SKIPPED=0  SURVIVED=11  SUSPICIOUS=1  TIMEOUT=0  cached=71  -> .moonbuggy/results.jsonl

That mutant is now KILLED, and cached=71 means the mutants in files you did not touch were not re-run. The second run is much faster than the first.

What you just did

You found a gap in a test suite that had full coverage of that line. Coverage said the line ran. It could not say whether anything checked the result.

That is the whole idea, and What mutation testing is is the ten-minute version of why it works.

Next

  • Not every survivor is a real gap. Some are impossible to kill — see Equivalent mutants.

  • The full output format, including the JSONL schema and jq recipes, is in Reading the output.

  • A realistic module walked end to end is in the Tutorial.