Closing the loop¶

Audience: you have read moonbuggy’s findings — or a program has — and you want the whole cycle in one place: find the gaps, hand them to something that fixes them, and prove the fixes worked. The consumer on the other end of the export can be a human with an editor or an automated agent; the loop is the same either way, because every step of it is a file format or a command with a frozen, versioned shape.

The loop in one line:

run -> export -> read survivors.jsonl -> fix tests -> run <id> -> read the verdict

Nothing in the loop touches a run’s artifacts: the export is a new file, and re-measuring one mutant leaves results.jsonl exactly as the full run wrote it. Each section below owns one leg.

The four commands¶

command	what it does in the loop
`moonbuggy`	produces the findings: one line per mutant in `.moonbuggy/results.txt`, one JSON record per mutant in `.moonbuggy/results.jsonl`
`moonbuggy export`	copies just the findings — every `SURVIVED` and `NO_COVERAGE` record — to `survivors.jsonl`, one JSON object per line
(your work)	strengthen or write a test, guided by the record’s `diff`, `nearest_test` and `survival_reason`
`moonbuggy run <id>`	re-measures that one mutant fresh and prints the verdict that answers “did the fix kill it?”

All of it works with no configuration file and no knowledge of moonbuggy’s internals. See Quickstart for getting the first run going.

Step 1 — export the findings¶

After a full run, moonbuggy export writes survivors.jsonl:

$ moonbuggy export
moonbuggy: exported 4 findings from 4 records to /tmp/loopdemo/survivors.jsonl

One JSON object per finding, and nothing else: killed, timed-out, suspicious and skipped mutants never export, so a zero-line file means “no findings” — not “something went wrong”. Check schema on the first line before parsing the rest, exactly as you would for any other moonbuggy output.

Step 2 — read each record¶

Every record carries the full results.jsonl envelope plus the export’s own fields. The ones that drive the fix:

field	what to do with it
`id`	the re-measurement input. Pass it straight to `moonbuggy run`.
`status`	`SURVIVED` (tests ran, none objected) or `NO_COVERAGE` (no test executes the line at all).
`diff`	what the mutant changed — the question your test has to answer.
`nearest_test`	for a `SURVIVED`, where to start reading; the tests that already execute the line.
`survival_reason`	why it survived, as one closed-vocabulary token: `no_coverage`, `covered_unasserted`, `logging_noise` (only under `--include-logging-mutants`), `accepted_equivalent` (a reviewed equivalent — leave it alone), or `null` meaning not yet classified. Never treat `null` as a reason.
`original`, `mutated`	the exact before/after line, so a program can reason about the change without re-deriving it.

The token set is frozen by the survival-reason vocabulary contract (survival-reason-v1.md); the export shape itself is frozen by survivor-export-v1.md. A consumer written against those versions reads survivors.jsonl unchanged.

Two records from a small module, one of each finding kind:

{"id": "calc.py:2:comparison_swap:0", "status": "NO_COVERAGE", "survival_reason": "no_coverage", "tests_run": 0, "mutated": "if value >= ceiling:", "diff": "- if value > ceiling:\n+ if value >= ceiling:", "nearest_test": null}
{"id": "calc.py:8:constant_int:0", "status": "SURVIVED", "survival_reason": "covered_unasserted", "tests_run": 1, "mutated": "return value + 2", "diff": "- return value + 1\n+ return value + 2", "nearest_test": "test_calc.py::test_bump_runs"}

(abridged for reading; real records carry every envelope field). The first says: no test executes calc.py:2 at all — write a test that reaches the line. The second says: the line runs, nothing checks its result — make an existing test assert on it.

Step 3 — fix the test, not the code¶

The record is a question about the tests, and diff is the question: if the code did this instead, would any test fail? A mutation testing finding is almost never a bug in the code — it is an untested behaviour. Strengthen a test or write one; do not “fix” the mutant.

For the NO_COVERAGE record above, the fix is a test that reaches line 2 at all. For the covered_unasserted one, the test called the function but never asserted — the same trap as a suite that “covers” a line by executing it:

# before: the line was executed, never checked
def test_bump_runs():
    bump(1)

# after: the behaviour is pinned
def test_bump_adds_one():
    assert bump(1) == 2

Step 4 — re-measure, and read the verdict honestly¶

moonbuggy run <id> measures the mutant fresh — never from the cache — with the same coverage pass, selection and runner a full run uses:

$ moonbuggy run calc.py:8:constant_int:0
KILLED    calc.py:8 constant_int line=8 nearest_test=- tests_run=1 killreason=assertion_failed id=calc.py:8:constant_int:0
moonbuggy: KILLED=1  re-measured=1 (.moonbuggy/results.jsonl is unchanged)

Exit code 0 when the mutant is killed, 1 when it is not, so the step works in a script. And because every exported id is valid input, the whole finding set replays at once:

$ jq -r '.id' survivors.jsonl | moonbuggy run -

For the machine loop — replay the whole finding set and get one JSON line per id that names what changed — add --trace-json --against:

$ jq -r '.id' survivors.jsonl | moonbuggy run - --trace-json \
    --against survivors.jsonl --flaky-probe 0

Each trace gains a reinject subdocument: the export record’s prior status and survival_reason, plus a transition token from a closed vocabulary. The token to gate on is survived->assertion_failed (or no_coverage->assertion_failed) — a real catch. The same table as above, in tokens:

survived->assertion_failed   fixed: the new test objects to the mutation
survived->killed_by_error    not verified: something crashed first
survived->survived           still a finding (silence is a fact about the new tests)

The stderr summary tallies the transitions and a caught=N that counts only the assertion_failed catches.

The one verdict rule to get right: killreason on the verdict line is what tells you the kill was real.

killreason=assertion_failed — a test’s assertion failed under the mutation. This is the only token that proves your new test actually checked the mutant’s behaviour. When you are verifying a fix, this is the transition to look for: the record was SURVIVED (or NO_COVERAGE), and the fresh verdict is KILLED with assertion_failed.
killreason=test_errored — a test errored rather than asserted. This status is KILLED_BY_ERROR, and it is not a kill to act on: the mutant broke the code badly enough that something raised before anything could object. It counts as a kill in the mutation score, but it says nothing about whether your tests check the mutated line. If your new test “kills” a survivor this way, you have not verified anything — check what actually happened before counting it as progress.

A consumer gates on the transition, not on the status keyword alone:

SURVIVED -> KILLED (assertion_failed)   fixed: the new test objects to the mutation
SURVIVED -> KILLED_BY_ERROR (...)       not verified: something crashed first
SURVIVED -> SURVIVED                    still a finding

The loop, checked¶

Every claim on this page is executed when the docs are tested, against a throwaway project. The export carries what a consumer needs:

>>> project = make_project({
...     "calc.py": "def bump(value):\n    return value + 1\n",
...     "test_calc.py": "from calc import bump\n\ndef test_bump_runs():\n    bump(1)\n",
... })
>>> _ = moonbuggy(cwd=project)
>>> proc = moonbuggy("export", cwd=project)
>>> proc.returncode
0
>>> findings = [json.loads(line) for line in (project / "survivors.jsonl").read_text().splitlines() if line]
>>> [(f["id"], f["status"], f["survival_reason"]) for f in findings]
[('calc.py:2:arithmetic_swap:0', 'SURVIVED', 'covered_unasserted'), ('calc.py:2:constant_int:0', 'SURVIVED', 'covered_unasserted')]

The fix is a real assertion, and re-measurement answers it:

>>> _ = (project / "test_calc.py").write_text(
...     "from calc import bump\n\ndef test_bump_adds_one():\n    assert bump(1) == 2\n")
>>> proc = moonbuggy("run", "calc.py:2:constant_int:0", cwd=project)
>>> proc.stdout.split()[0], proc.returncode
('KILLED', 0)

The verdict line carries the only killreason that proves the test checked the mutant:

>>> "killreason=assertion_failed" in proc.stdout
True

And every exported id is valid input — the whole set re-measures in one command:

>>> proc = moonbuggy("run", *[f["id"] for f in findings], cwd=project)
>>> sorted(line.split()[0] for line in proc.stdout.splitlines()
...        if line.split()[0] in ("KILLED", "SURVIVED", "NO_COVERAGE"))
['KILLED', 'KILLED']

(Only two findings remain in this tiny project; a real export may also contain NO_COVERAGE rows, which run - re-measures and gates on exactly the same way.)

Where to go next¶

Reading the output — the full JSONL schema, the plaintext line, and every field the export envelope carries.
Equivalent mutants — survivors that no test can kill, and the ledger that records the human decision to accept one.
Troubleshooting — when a run itself is lying to you.