Test coverage metrics worth putting in a test strategy document are the ones whose gaming mode you can live with: defect escape rate segmented by stage, coverage of changed lines, flake rate, and mean time to detect. Total line coverage, test count, pass rate on a suite nobody trusts, automation percentage, and execution time all read like progress and can be satisfied without the product getting any safer. Report the first four. Cut the rest.
You're filling in the metrics section of a test strategy document, and someone with a VP title is going to read it. Every listicle of test coverage metrics gives you the same unhelpful abundance: twenty numbers, no ranking, no sign of which four survive a leadership review. Pick the wrong four and you've quietly told your team what to optimize for next quarter, because once a number appears in a report someone owns, people start moving it.
This is written for a team shipping a web product on a continuous release cadence, deciding what to report because a person is asking, not a regulator. In a regulated or safety-critical context, a coverage floor is a compliance artifact you hit regardless of what it signals, which is a different document than this one.
The 4 test coverage metrics worth reporting
Four metrics survive the question that matters: what happens once your team knows the number is read upward. Each earns its place because the cheapest way to move it is also a real improvement, not a shortcut around one. These four are worth reporting independent of whether you also gate on them; test entry and exit criteria covers using some of the same numbers to block a release, a related but different job than reporting them to a VP.
Defect escape rate, segmented by stage
Defect escape rate tells you where your net has holes, not how big the net is. Segmented by the stage that caught it, unit, integration, staging, or production, it turns into direction: a rising share of production catches means your earlier stages are failing, not the product. Defect escape rate covers the definition in full; here it's enough to know it belongs in the report, segmented, not as one blended number.
Coverage of changed lines
Coverage of changed lines is coverage scoped to the diff, not the repo. It answers whether the new risk is tested, not whether the old code is, and because it's scoped to what changed this release, it moves when behavior moves. A repo-wide percentage can sit flat for a year while entire features ship untested; changed-line coverage can't hide that.
Flake rate
Flake rate is the percentage of runs where a test changes its verdict with no code change behind it. Report this one or don't bother reporting the other three, because every other number assumes the tests underneath it are telling the truth. A defect escape rate computed against a suite nobody trusts is measuring theater, not risk.
Mean time to detect
Mean time to detect is how long a defect lives, from the commit that introduced it to the first signal, test, alert, or bug report, that catches it. It ties the section to delivery speed instead of leaving it as its own island, and it's the natural bridge to why testing itself is often the bottleneck behind a slow DORA deployment frequency. It's also the cleanest of the shift left testing metrics here: it rewards catching a defect one stage earlier, which is the entire premise of shifting testing left.
Position on the axis is the cheapest way to move the number: the five on the left can be satisfied without the product getting safer, and the four on the right can't be.
The 5 coverage metrics that get gamed
The other five test coverage metrics show up in nearly every QA report template, and each fails the same test: the cheapest way to move it doesn't require the product getting any safer.
Total line coverage is the most familiar offender. A generator meets any target you name, executing every branch and asserting nothing that would fail on a real bug. The number can hit 90% and mean nothing about whether the code works.
Test count rewards quantity over signal. A suite doubles in size and halves in signal, the report shows growth, and the product doesn't get any safer; the two facts sit right next to each other, unremarked.
Pass rate on a suite nobody trusts measures the suite's tolerance, not the product's health. Fill a suite with weak assertions and it reports 100% forever, because a test that checks almost nothing almost never fails.
Automation percentage rewards automating whatever was easiest. The flows that look best under that incentive are simple and low-risk; the complex flows that actually matter are the last to get covered and the first to matter when something breaks.
Test execution time starts as a legitimate engineering concern and becomes a vanity metric the moment it lands in a strategy document, because the cheapest way to bring it down is deleting or skipping slow tests, and slow tests are disproportionately the ones that catch the escapes the shortlist above is trying to measure.
None of these five belong in your four slots, though several earn a place on a broader QA metrics dashboard or automation metrics that predict release quality. A report and a dashboard tolerate different amounts of noise.
How each coverage metric gets gamed
The same nine metrics, side by side: what each measures, how a team games it, and whether it belongs in the report.
| Metric | What it measures | How it gets gamed | Report it? |
|---|---|---|---|
| Defect escape rate by stage | Where the net has holes | Reclassify severity to dodge the count | Yes |
| Coverage of changed lines | New risk under test | Pad the diff with trivial refactors | Yes |
| Flake rate | Whether the suite can be trusted | Quarantine flaky tests, hide them from the count | Yes |
| Mean time to detect | How long a defect lives unnoticed | Widen "detected" to include vague alerts | Yes |
| Total line coverage | Lines a parser touched | Generate tests that assert nothing | No |
| Test count | Suite size | Split one test into ten thin ones | No |
| Pass rate, untrusted suite | The suite's own tolerance | Write weak assertions that never fail | No |
| Automation percentage | Share of manual work removed | Automate the easy flows, skip hard ones | No |
| Test execution time | CI runtime cost | Delete slow integration and E2E tests | No |
How Autonoma measures coverage in flows, not lines
Every vanity metric in that table fails the same way: it measures the shape of the source code or the suite rather than the behavior of the product, which is precisely the gap a generator can satisfy without the product getting any safer.
Autonoma's coverage number is built to resist that gap by changing the denominator. Connect a codebase and the Planner agent reads routes, components, and flows to derive user journeys instead of source lines: sign up, add to cart, check out, invite a teammate.
The Executor runs each journey against a live preview, the Reviewer classifies what happens (a real bug, an agent error, or a plan mismatch), and the Diffs Agent updates the set of covered flows on every PR by reading the code diff.
The unit being counted is a flow a person can actually perform, not a line a parser can count, and that's what makes the number resistant to the specific gaming modes in the table above: you cannot satisfy "the checkout flow is covered" with a test that imports a module and asserts nothing, the way you can satisfy a line-coverage percentage target.
Both tracks describe the same six areas of one codebase. Changing the denominator from lines to flows is what makes checkout and invite teammate show up as gaps at all.
This sits closest to coverage of changed lines on the shortlist above, not as a replacement but as a stricter version of the same idea: instead of asking whether the changed lines have any test touching them, it asks whether the changed user-facing behavior has a passing journey behind it. It has nothing to say about defect escape rate or mean time to detect directly; both still depend on your monitoring and triage process, not on how coverage happens to be denominated.
Is line coverage a good quality signal?
Line coverage is a poor signal of test quality regardless of who or what wrote the tests, an argument made in full elsewhere: see why coverage misleads for AI-generated tests, assertion coverage versus line coverage, and mutation testing versus code coverage.
What goes in your test strategy document
Four test coverage metrics, stated with their gaming mode next to them, is a stronger reporting section than twenty numbers with none. Defect escape rate by stage, coverage of changed lines, flake rate, and mean time to detect each fail informatively: if one moves the wrong way, you know roughly what broke and why. That's worth more to whoever reads this document than comprehensiveness ever was.
The four numbers you write into that section today will outlive the document itself. Six months from now nobody rereads the strategy doc, but the team is still optimizing for whatever got reported, because that's what reporting does. That's Goodhart's law in miniature: once a measure becomes the target, people optimize the measure instead of whatever it was meant to represent.
One number on the shortlist is worth watching for a different reason: flake rate tends to get more expensive to keep low as a suite grows, unless something is actively separating a real regression from an agent error or a stale assertion on every run. That's the layer where Autonoma's Reviewer agent does its job, classifying each failure before it ever reaches your report, so the flake number you write down keeps meaning "the suite can be trusted" instead of drifting upward while nobody's headcount for test maintenance grows to match. Whichever four numbers you land on, the rule underneath all of them is the same: the ones that survive are the ones somebody keeps paying attention to after the document ships.
Frequently Asked Questions
Four metrics survive the incentive test: defect escape rate by stage, coverage of changed lines, flake rate, and mean time to detect. Coverage is scoped to the lines changed in the current release rather than the whole codebase, and flake rate is the percentage of runs where a test flips verdict with no code change behind it. Each one fails informatively when it moves the wrong way, and each one is expensive or pointless to fake, which is what a number in a document someone reads upward actually needs.
Raw line coverage across the whole codebase is a weak metric because a test suite, generated or hand-written, can hit any percentage without asserting anything meaningful about behavior. It measures the shape of the source, not whether the product works. Coverage scoped to the lines changed in the current release is a stronger, narrower version of the same idea, and mutation testing and assertion-density checks go further into why the raw percentage misleads, covered in depth elsewhere on this site.
Coverage of changed lines measures whether the specific lines modified, added, or removed in the current release have any test executing them, rather than measuring coverage across the entire codebase. It answers whether the new risk introduced this cycle is under test, which a repo-wide percentage can't do, since that number stays flat while entire features ship untested elsewhere in the code.
A QA report for a leadership audience should hold to a small number of metrics that each fail informatively, not a long list nobody ranks. That means something on where defects escape by stage, something on whether new risk is tested (coverage of changed lines), something on whether the suite itself can be trusted (flake rate), and something tying testing to delivery speed (mean time to detect). Twenty metrics with no ranking gives the reader nothing to act on; four with stated gaming modes gives them a document they can actually use.
Flake rate is the single best proxy: the percentage of test runs where a result flips with no underlying code change. A suite with a high flake rate can't lend credibility to any other number in the report, including defect escape rate and coverage, because a green or red result from an untrustworthy suite isn't evidence of anything. Pass rate alone doesn't measure this, since a suite full of weak assertions can report 100% forever.
Autonoma changes what coverage measures rather than replacing it as a metric, counting user flows the Planner and Executor agents run instead of source lines. Those flows run against a live preview, so the number in your report is denominated in behavior instead of source shape. It doesn't touch defect escape rate or mean time to detect directly, and it isn't a substitute for deciding your own flake-rate ceiling; those still depend on your triage process and your suite's own reliability.
Test coverage metrics split into statement, branch, function, path, condition, requirements, and flow coverage, each counting a different unit of a codebase or a product. Statement coverage counts executed lines, branch coverage counts decision outcomes, function coverage counts functions called, path coverage counts execution paths, condition coverage counts boolean sub-expressions, requirements coverage counts requirements exercised, and flow coverage counts user journeys with a passing test.
Test coverage is measured as covered units divided by total units, where a unit can be a line, a branch, or a user flow. The arithmetic never changes; the real decision is which denominator you pick.
Code coverage is denominated in source units a parser can see, like lines or branches; test coverage is the broader term and its denominator isn't fixed. Test coverage can be denominated in requirements exercised or user flows completed, neither of which a parser can compute from source alone.
