Smoke test metrics quantify whether a smoke gate is actually catching something, not just running on schedule. The four that matter are smoke pass rate, escaped-defect rate past the gate, time-to-signal, and suite size measured against a declared critical-path count. None of the four is useful as a name alone; each one only means something once you can show the formula and the worked number behind it.
Someone asked whether the smoke gate is actually catching anything, or how many smoke checks a suite should have, and the honest answer requires arithmetic you can show your reader, not a metric name copied off a vendor's KPI page. Most pages on smoke test metrics list five or six names and stop, with no formula, no worked number, and no flag for which of those numbers quietly goes up while the suite gets worse.
If you need the definition first, that lives on what smoke testing actually checks; this page assumes you already have it.
This is written for the engineer, SDET, or release owner right-sizing or defending an existing smoke gate: proving it works, sizing it, or explaining the check count to someone above them. It's not written for the QA lead building a suite-allocation strategy across a test org, the reader asking whether an AI-generated suite tests anything real (that's ai-generated test theater), or someone picking edge-case inputs for a single test case, one level of detail below anything here.
Smoke Test Pass Rate: The Metric Everyone Reaches for First
smoke test pass rate = (passing smoke runs / total smoke runs) × 100
Take one week of runs against main: 120 smoke runs triggered, 111 came back green. 111 / 120 = 0.925, which is a 92.5% pass rate.
That number is easy to compute and easy to put on a dashboard, which is exactly the problem. Smoke test pass rate is a build-health metric, not a gate-quality one. A gate that never goes red is equally consistent with two different worlds: builds that are genuinely healthy, and a suite checking so little it could not fail if it tried. The number alone can't tell you which; it needs the other three metrics below to mean anything.
Escaped-Defect Rate: What Got Past the Gate
escaped-defect rate = (defects that passed smoke and surfaced later / total defects found in the release window) × 100
One release window surfaced 43 defects. Six were critical-path breakages the smoke gate passed clean, only showing up downstream. 6 / 43 = 0.1395, rounding to a 14.0% escaped-defect rate.
That headline percentage is the one people quote, but it isn't worth tracking week over week, because the denominator (total defects found) moves for reasons that have nothing to do with the smoke gate. Two other denominators survive that problem, because they are business-side counts instead of suite-side ones: escapes per release, 6 escapes / 4 releases = 1.5 escapes per release, and escapes per critical flow, 6 escapes / 12 critical flows = 0.5 escapes per flow. For the general-purpose version of this metric, the defect escape rate breakdown covers it without re-deriving it here. The distinction behind why those two survive is the same one that breaks pass rate later in this article.
Suite-side counts move when you add tests; business-side counts do not.
Time-to-Signal: How Long Until You Know
time-to-signal = median wall clock from commit pushed to smoke verdict posted
Sum the components for a typical run: runner queue wait, 40s. Build and deploy to preview, 3m 10s (190s). Suite execution, 2m 05s (125s). Report and annotate, 15s. 40 + 190 + 125 + 15 = 370 seconds, which is 6 minutes 10 seconds.
Time-to-signal earns its place because it's the one metric here that visibly worsens as a suite balloons, a useful counterweight to pass rate, which can hold steady or even improve while the suite grows. Martin Fowler's continuous integration write-up sets a ten minute rule for build times, on the grounds that every minute cut from the build is a minute back for every developer on every commit. Breaking it into components matters because the fix differs for each: queue wait is a capacity problem, build time is an infrastructure problem, and suite execution is a scope problem, the one component that grows in direct proportion to how many checks you added.
Queue wait, build, execution and reporting add up to 6 minutes 10 seconds, and each segment has its own fix.
Suite Size Against Critical-Path Count: How Many Smoke Tests Should You Have
critical-path coverage = (critical paths with at least one smoke check / total declared critical paths) × 100
check density = total smoke checks / declared critical paths
Twelve declared critical paths. The suite holds 34 checks. Eleven of the twelve paths have at least one check. Coverage: 11 / 12 = 0.9167, which is 91.7%. Density: 34 / 12 = 2.83 checks per path.
This is the actual answer to "how many smoke tests should I have," and it isn't a magic number. The target check count is set by the declared critical-path count, at roughly one level of depth per path, the same shallow-not-deep property that separates a smoke check from a regression case. Density between 1 and 3 checks per path reads as a smoke suite behaving like one. Density past roughly 6 signals the suite has quietly grown into a regression suite wearing a smoke label, worth splitting the two apart before it starts eating the time-to-signal budget. That's a rule to reason through for your own suite, not a benchmark to hit blindly.
Eleven of twelve critical paths are covered; the twelfth is the gap coverage alone would hide.
| Metric | Formula | Worked result | What it hides |
|---|---|---|---|
| Smoke test pass rate | passing runs / total runs × 100 | 92.5% (111 / 120) | Build health, not gate quality |
| Escaped-defect rate | escapes past gate / total defects × 100 | 14.0% (6 / 43) | Denominator you don't control |
| Time-to-signal | commit push to verdict, wall clock | 6m 10s (370s) | Nothing, it gets worse honestly |
| Critical-path coverage | covered paths / total paths ×100 | 91.7% (11 / 12) | Depth per path |
| Check density | total checks / critical paths | 2.83 checks per path | No universal target, just a range |
How Autonoma Reports Smoke Signal
Every metric above assumes a suite you can point to and count: this many runs, this many checks, this many declared paths. That holds for a hand-curated suite. It gets shakier once the suite is generated and regenerated on every change, because a check count that resets every pull request isn't a stable thing to graph.
That's the situation our agents create on purpose. They read the customer's codebase, derive the critical paths the way a human reviewer would, and generate and maintain end-to-end tests that run against the live, deployed application, not a saved snapshot. The Diffs Agent re-reads every pull request and updates the suite to match: new paths picked up, dead paths retired. Because the suite is regenerated rather than curated, the suite-side denominators from the table above stop describing anything stable enough to track. The number worth surfacing instead is the review-bound one: how many failures actually need a person to look at them, where the signal concentrates once authoring stops being the bottleneck.
Put plainly: this gate used to be curated by hand because a human had to run each check. When checks are derived from the codebase and healed automatically as it changes, the gate stops being a list and becomes a signal you regenerate, and the scarce work moves from writing checks to reviewing what the suite surfaces.
That's one piece of the picture, not the whole one. Autonoma is the behavioral, end-to-end execution layer. It doesn't replace a unit test runner, a contract-testing framework, a load-testing tool, an accessibility scanner, or your incident data, and was never meant to. Escaped-defect counting depends entirely on your own defect tracker and your own definition of a critical flow. No tool, including this one, can supply that judgment for you.
The Honest Flag: Why Pass Rate Breaks First When Suites Balloon
Raw pass rate is the metric that breaks first when a suite grows, and the arithmetic proves it. A small, curated suite runs 34 checks with 3 failing: 31 / 34 = 0.9118, a 91.2% pass rate. A ballooned, machine-generated suite runs 340 checks with 9 failing: 331 / 340 = 0.9735, a 97.4% pass rate.
The pass rate improved by more than six points while the absolute number of failing checks tripled. The second suite isn't healthier than the first; it's just bigger, and its denominator grew faster than its numerator problem did.
The failing segment shrinks visually even though the absolute failure count tripled.
That's the rule that survives a machine-authored suite: a metric whose denominator is a suite-side count (checks, runs, cases) can be improved just by adding tests, whether or not the product got any healthier. Metrics whose denominator is a business-side count (releases, critical flows, incidents) can't be gamed the same way, which is exactly why escapes per release and escapes per critical flow belong on a dashboard instead of raw pass rate. This is not a smoke-specific quirk. DORA's delivery metrics measure instability with change fail rate, the ratio of deployments that require immediate intervention, so its denominator sits outside the test suite entirely. DORA's own list of pitfalls opens with setting metrics as a goal and ignoring Goodhart's law. For the false-confidence version of this problem in AI-generated suites, ai-generated test theater covers it directly; for what to actually display once you've picked the right denominators, the QA metrics dashboard guide covers that side of it.
Getting These Numbers Onto a Release Gate
None of these four metrics is useful alone. Pass rate says the build shipped green. Escaped-defect rate, counted against releases and critical flows, says whether the gate catches what matters. Time-to-signal says what the gate costs every run. Wiring those numbers into a job that actually blocks a promote is covered in the automated smoke gate walkthrough. Coverage and density answer how big the suite should be, without picking a number out of the air. Together they feed into whether a build is ready to ship, which a release-readiness checklist walks through as its own gate.
If your smoke suite is still a list someone wrote by hand three years ago, most of this article is diagnostic. Autonoma is built for the version of this problem where the suite regenerates itself instead of quietly going stale, so these metrics measure something still true next quarter, not a snapshot of what critical paths looked like the last time someone remembered to update the list.
Frequently Asked Questions
Smoke test metrics are the numbers that show whether a smoke gate is catching something rather than just running. Four carry weight: pass rate, the share of smoke runs that come back green; escaped-defect rate, the share of defects the gate passed clean that surfaced later; time-to-signal, the wall clock from commit pushed to verdict posted; and suite size measured as coverage and density against a declared critical-path count. Each one needs a formula and a worked number behind it, because the names alone do not tell you whether the gate works.
There's no fixed number to target. The right count is set by your declared critical-path count, at roughly one check per path, which gives a check density between 1 and 3. A suite with 12 critical paths and 34 checks has a density of 2.83, which reads as a smoke suite doing its job. Density climbing past roughly 6 usually means the suite has grown into a regression suite wearing a smoke label, and it's worth splitting the two apart rather than adding more critical paths to justify the size.
There isn't a target percentage worth chasing on its own, because pass rate is a build-health metric, not a gate-quality one. A suite with 92.5% pass rate could be catching real problems or could be checking so little it can barely fail. Pass rate only becomes meaningful alongside escaped-defect rate and check density, which show whether the suite is actually watching the paths that matter.
Because pass rate's denominator is a suite-side count: the total number of checks or runs. Adding tests grows that denominator faster than it grows the numerator problem, so the percentage improves even while the absolute number of failures grows. A 34-check suite with 3 failures reads as 91.2%; a 340-check suite with 9 failures, triple the failures, reads as 97.4%. Metrics with a business-side denominator, like escapes per release, don't have this problem.
Autonoma works on the thing those metrics measure, which is usually where the numbers actually move. It generates and maintains the smoke suite itself, deriving critical paths from your codebase and running checks against a live preview, with the Diffs Agent updating the suite on every pull request. That is what pulls escaped-defect rate down and holds check density in range, because a suite that tracks the product catches what a stale one passes clean. You keep reading the four metrics in your own CI reporting and defect tracker, against your own definition of a critical flow, and Autonoma keeps the suite behind those numbers current.
The execution component does, because it scales roughly linearly with how many checks run. Queue wait and build time don't scale with suite size at all; they're capacity and infrastructure problems respectively. That's the reason to break time-to-signal into its four components instead of tracking one wall-clock number: the fix for a slow signal depends entirely on which component grew.
