Manual smoke testing is the right call when a build's shape changes weekly, when there's no CI pipeline to run a gate automatically, or when the suite is small enough that a person clears it in a couple of minutes. The decision comes down to one comparison: cumulative manual cost (runs per week times minutes per run times weeks) against one-time authoring cost plus ongoing maintenance. Below that break-even point, keep it manual.
Every team running a smoke gate eventually asks the same question out loud: why is a person still doing this by hand? Most weeks, nobody has done the arithmetic. They've done the vibe check instead: automation feels more mature, so the manual pass gets treated as a thing to graduate out of, on a timeline nobody derives.
That's backwards. The smoke gate is the cheapest, most frequently run check in the whole release process, which is exactly why a small automation decision compounds fastest here, in either direction. Get it wrong and you're either paying an engineer to click through the same five screens every Tuesday for the next three years, or paying to author and maintain a suite that a person could have cleared in the time it took to open the pull request.
This is for the engineer, SDET, or release owner deciding whether today's build ships, or asked to name and staff a gate. It's not for the QA lead writing a suite-allocation strategy, and it's not the "are my AI tests real" question. Narrower than either: one gate, one arithmetic decision.
Three situations where manual smoke testing wins
Smoke testing is the shallow, wide check that runs first, before anything else, to confirm a build is alive enough to bother testing further, distinct from the deeper sanity and regression gates that follow it. Whether that check runs by hand or by machine is a separate question, and for a specific set of teams, manual is the correct answer, not the fallback.
The app changes shape weekly. If the login screen, the checkout flow, and the primary navigation are still being redesigned sprint over sprint, any suite authored against today's screens targets something that will not survive to the second run. The full authoring cost gets paid every single time, which means it never amortizes. A person looking at whatever happens to be on screen this week costs the same two minutes it always did.
There's no CI to hang the gate on. An automated check nobody runs automatically isn't automation, it's a manual check with a maintenance bill attached. If there's no pipeline stage where a smoke suite would actually block a bad promote, writing one buys a script somebody still has to remember to run by hand. Fix the pipeline gap first.
The suite is five checks a competent person clears in two minutes, run once a week. This is the case worth actually computing, because the arithmetic is small enough to do in your head and the instinct to automate anyway is strong. Which five checks belong in that suite is its own question, covered in how to pick smoke test cases. Run the numbers below and the five-check suite loses the case for automating almost every time.
Only the last question is arithmetic; a missing pipeline or churning screens settle it first.
The cost rule, derived not asserted
Strip away the vibes and the comparison is exactly two numbers running against each other over time.
Manual cost accumulates every run: runs per week, times minutes per run, times weeks. Automated cost has a different shape, not just a different size: a one-time cost to author the checks, plus an ongoing cost to keep them passing as the app changes underneath them. That ongoing term, maintenance, is the one most back-of-envelope comparisons quietly drop, and it's usually what decides the real cases, the same reason even Google continually invests in keeping its own suites trustworthy rather than treating a passing suite as a one-time achievement. We've broken down what that upkeep actually costs over time separately, since it rarely shows up in a first estimate.
| Cost component | Manual pass | Automated suite |
|---|---|---|
| Per run | Minutes per run x rate | None, already authored |
| One-time | None | Authoring hours x rate |
| Weekly, ongoing | Runs x minutes x rate | Maintenance minutes x rate |
| Over the horizon | Runs x minutes x weeks x rate | Authoring plus (upkeep x weeks x rate) |
Set the two totals equal and solve for weeks: break-even equals authoring cost, divided by rate times the weekly gap between manual cost and maintenance cost. If runs times minutes is smaller than upkeep minutes, the automated suite's weekly maintenance alone costs more than the manual pass did, so there's no break-even at any horizon. Automating stays more expensive than manual every week, forever, which is precisely the trap the five-check case above sets.
Ten checks, thrice weekly: a smaller authoring term slides break-even from week 40 back to the first month.
Three scenarios, worked
Assume a loaded rate of $60 an hour, $1 a minute (illustrative, not an industry figure), across three suites over a 52-week horizon. These are numbers we constructed to check the shape of the rule, not industry data, so run your own numbers through the same formula.
Five checks, one run a week, two minutes total: manual cost is 1 x 2 x 52 minutes, $104 a year. Automating means four hours of authoring ($240) plus five minutes a week of upkeep as the UI shifts under it ($260 a year). Automated total: $500. Manual wins by roughly five to one, exactly the case above: authoring never earns out.
Fifteen checks on every merge, 20 runs a week (a realistic deployment frequency for a team merging several times a day), 15 minutes a run: manual cost is 20 x 15 x 52 minutes, $15,600 a year. Authoring 20 hours of checks costs $1,200; upkeep at 20 minutes a week adds $1,040. Automated total: $2,240. Automating wins by about seven to one, and would have broken even in under five weeks ($1,200 divided by $280, about 4.3 weeks of runs).
Ten checks, three runs a week, 10 minutes a run: manual cost is 3 x 10 x 52 minutes, $1,560 a year. Authoring ten hours costs $600; upkeep at 15 minutes a week, since this UI moves more, adds $780. Automated total: $1,380. Automating wins, but only by $180 on the year, with break-even at week 40. Run the same suite for 30 weeks instead of 52 and manual is still cheaper, $900 against $1,050, worth re-checking with your own numbers before committing.
| Scenario | Manual cost, year | Automated cost, year | Verdict |
|---|---|---|---|
| Five checks, weekly | $104 | $500 | Manual wins |
| Fifteen checks, every merge | $15,600 | $2,240 | Automate wins |
| Ten checks, thrice weekly | $1,560 | $1,380 | Automate, barely |
A tiny weekly suite stays manual; a per-merge suite flips to automate; the ten-check case is a coin toss.
How Autonoma changes the automate-or-not math
All of that arithmetic assumes authoring and maintenance stay at their old-world prices: a person writing the suite by hand, then patching it every time a selector moves. That assumption is exactly what made the five-check suite and the weekly-reshape case correctly manual above. It's also the assumption generation is built to break.
We built Autonoma's agents to read the codebase directly, plan test cases from the routes and components that actually exist, and run them against a live preview of the application before a reviewing agent looks at the results. Because the suite is derived from the code instead of typed in by hand, and because it heals itself against the running application as the code changes instead of waiting for someone to notice a broken selector, a smoke gate stops being a hand-curated list a person maintains and becomes a signal you regenerate. The scarce work moves from writing and patching checks to reviewing what the suite surfaces.
Autonoma shrinks the authoring and maintenance terms, which is what pulls break-even earlier.
What doesn't change
Two of the three manual-wins cases above are exactly the cases this shift is supposed to move. The third isn't, and it's worth being precise about which is which.
The no-CI case survives untouched. No amount of cheap, self-healing test generation conjures a pipeline stage to hang a gate on. If merges don't trigger anything, the problem was never the cost of writing checks, it was the absence of a place to run them automatically. Fix that first, or keep running the pass by hand.
The five-check, once-a-week case is the one most likely to flip. Run the same formula with authoring priced at the cost of connecting a repository instead of four hours of a person's time, and with maintenance mostly absorbed by a suite that heals itself against the running application instead of someone patching a selector, and the automated total for the year can fall well under what the manual pass costs. The case that correctly favored manual under the old numbers stops being clear-cut once the authoring term shrinks that far.
The weekly-reshape case is the genuinely hard one, and it's where judgement stays human either way. If the screens are being redesigned because product decisions haven't been made yet, no generator, ours or anyone else's, can plan tests against a shape that doesn't exist yet. What changes is how much churn a suite can absorb once the shape stabilizes even a little: partial stability that used to still lose the arithmetic can now clear the bar, because re-deriving checks each time the shape shifts costs closer to connecting the repository again than to a person rewriting a suite from scratch. What doesn't change is who decides whether a failing check is a real regression or an intended change, and who decides what counts as release-blocking in the first place. That call was never arithmetic, and it still isn't. For where that boundary sits, and for the parts of QA that stay a deliberate human judgement call regardless of how cheap generation gets, see our take on manual QA next to AI testing.
Deciding for your own gate
None of the numbers above are yours. That's deliberate. Plug in your own runs per week, your own minutes per run, your own authoring estimate, and your own honest guess at weekly maintenance, and the same formula tells you where your smoke gate actually sits, not where an industry average says it should sit.
If the answer comes back manual, keep it manual, and don't feel behind for it. That's what the arithmetic is for. If the answer comes back automate, or if you're not confident your maintenance estimate is honest (nobody's is, the first time), that's the point where it's worth connecting a repository to Autonoma and running the same math again with a much smaller authoring term on one side of it. The gate doesn't stop being a decision you make. It just stops being one you have to re-derive by hand every time the app changes shape.
Frequently Asked Questions
Multiply your runs per week by your minutes per run, then multiply by the weeks you care about. Compare that to a realistic authoring estimate plus weekly maintenance over the same horizon. If the manual total stays lower across a full year, not just this week, it should stay manual until the suite grows or the run frequency rises.
Dropping the maintenance term. Most rough comparisons only price the one-time cost of writing the automated checks against the manual pass, and skip the ongoing cost of keeping those checks passing as the UI changes. Maintenance, not authoring, is usually what decides a close case.
Yes, in one specific direction: as runs per week or minutes per run go up, the manual side of the comparison grows without bound while the automated side only grows by its maintenance term. A suite that correctly stayed manual at low frequency can cross the break-even the moment it starts running many times a day.
More often than the old arithmetic suggests. Connecting a repository replaces the hours of hand-authoring this article prices in, and the suite then heals itself against the running app instead of needing someone to patch selectors, so even a five-check suite's break-even can move in Autonoma's favor. And once you're already running Autonoma for your larger suites, folding the five-check gate into it costs almost nothing extra. The one case where staying manual is still clearly right is a suite that genuinely runs once a week and nothing else, and most teams cross that line sooner than they expect.
Generation cannot plan tests against screens that don't exist yet, so a suite in the middle of a real redesign stays a judgement call for a human to author and re-author by hand until the shape settles. Once the redesign stabilizes, even partially, re-deriving the suite from the codebase becomes cheap again.
