Automated smoke testing wires a shallow, four-check suite into a CI/CD pipeline as a blocking gate: a job that runs after the build and before promotion, and fails the deploy outright on a red result instead of leaving a human to notice later. The gate lives in the workflow file itself, with an explicit exit-code contract and a job dependency that structurally stops a bad build from reaching production.
Most teams that call their smoke suite a "gate" are wrong about what makes it one. The suite runs. It reports red or green. The deploy ships anyway, because nothing in the pipeline was ever told to listen to that result.
The fix isn't a better suite. It's three unglamorous pieces of CI wiring that decide whether a failure actually stops anything: where the check sits relative to build and promote, a job dependency that makes the next stage unreachable on a red run, and an exit-code contract that tells a real bug apart from a broken test runner.
This is written for the engineer, SDET, or release owner who already has, or is about to write, a smoke suite and needs it wired into CI as something that actually blocks. It is not written for someone deciding which paths belong in that suite: a copyable suite and checklist already owns that decision. It's also not for someone building the metrics that prove the gate is catching anything, or someone still choosing between smoke, sanity, canary, and regression. This page assumes the suite exists and answers one question: how does it become a gate instead of a script somebody remembers to run before a release.
That distinction, between a gate and a check that merely runs, starts with where in the pipeline the smoke job actually sits.
Where Automated Smoke Testing Sits in the CI/CD Pipeline
A pipeline has more than one place a smoke check could live, and only one earns the name gate. Some teams run it after deploy, against production traffic, once the promote already happened. That's a useful early warning, but by the time it fires the bad build is already live, and the check has become a smoke alarm instead of a smoke test.
The position that earns the name is post-build, pre-promote. The gate runs against the exact artifact that just built, deployed to a preview environment that is a real, running candidate, and it runs before promotion to the next environment, not after. That ordering is what makes it blocking instead of advisory, and it matters more than the checks themselves. If the smoke job can fail and the promote job runs anyway, nothing downstream actually changed, and the gate was theater.
One more distinction is worth drawing here, since it gets confused with this one. A check that runs against a slice of real production traffic after a candidate is already partially live is a different discipline with a different name, canary testing. Smoke asks whether a candidate deserves to be promoted at all. Canary asks whether a candidate already serving some of production is safe to receive the rest. Confusing the two is how a team ends up running an expensive canary rollout against a build a five-minute smoke gate would already have caught.
Post-build, pre-promote: the gate sits between a real candidate and the step that ships it.
The Gating Job: A Real CI Workflow
Here's the workflow that makes post-build, pre-promote structural instead of a convention someone has to remember. Three jobs: build compiles the application and deploys a preview candidate, smoke runs the suite against that exact candidate, and promote only fires if smoke succeeded, wired through GitHub Actions' job dependencies rather than a comment telling the next engineer to check manually.
The promote job lists build and smoke as dependencies, which means GitHub Actions will not even attempt to run it unless both upstream jobs report success. There is no code path where a red smoke job produces a promote. The always-upload condition on the artifact step matters just as much as the gate itself. Without it, a failing smoke job would also swallow its own evidence, and the report that would tell someone what actually broke never gets uploaded.
The smoke job's real content is a call to bash scripts/run-smoke.sh, wrapped in a few lines that capture its exit code and annotate the run summary before the step fails, and that script is where the exit-code contract actually lives.
set -euo pipefail matters here for a boring but important reason. Without it, a failed step inside the script can get silently swallowed and the script exits clean anyway, which is the one failure mode that defeats a blocking gate completely. A false-green smoke job is worse than no smoke job at all, because it promotes with confidence instead of without it.
The suite the script calls is deliberately narrow: four checks, one level deep, the same shape worked through in the storefront example, this time as runnable Playwright instead of a table.
The Exit-Code Contract
| Exit code | Meaning | Pipeline action |
|---|---|---|
0 | All four checks passed | Promote job runs |
1 | A real check failed | Promote job blocked |
2 | Harness or infra error | Promote blocked, annotated as harness error |
Three exit codes, and the distinction between 1 and 2 is the one most hand-rolled gates get wrong. A 1 means the suite ran clean and a real check failed: the login form rejected valid credentials, or the catalog returned zero results again. A 2 means the harness itself never got a fair shot: the preview environment didn't come up in time, or the browser binary failed to launch. Collapsing both into the same block-the-deploy bucket is safe for the pipeline but expensive for the humans on call, since every 2 treated as a 1 sends someone hunting for a product bug that doesn't exist.
Both codes still fail the smoke job and leave the promote job unreachable through the same job dependency, a 2 never gets a free pass to production. What differs is the signal an on-call engineer sees when they open the run: the smoke step captures the script's exit code and writes a distinct annotation for each case, so a 1 reads in the run summary as a real check failure and a 2 reads as a harness error, before anyone starts debugging either one. Running Playwright inside GitHub's runners has more if the harness-flake side is what's biting you.
The same script, three exit codes, three different pipeline responses.
Blocking vs Advisory: What Makes a Gate Real
Nothing stops a team from wiring a smoke job into CI and never letting it touch the promote step: run it, post the result to a channel, let a human glance at it before deploying anyway. That's a real thing to build, and it has a name: an advisory check. It is not a gate. Letting the smoke step continue on error, or writing a promote job with no dependency on smoke at all, both produce a pipeline that runs the suite and ships regardless of what it found. Nothing downstream of it changed.
If you're deciding whether to name and run a gate for your own team, this is the actual decision, not the checks themselves: does a red result structurally stop the next step, or does it just get logged somewhere a human might read later. Everything else, which four checks, what the YAML looks like, is implementation.
This matters even more on a build a coding agent shipped without a human reading every line. A blocking quality gate matters more, not less, when the distance between commits moved fast enough that nobody fully reviewed it.
Same two jobs, different wiring: one dependency is the entire difference.
How Autonoma Keeps the Smoke Gate Green as Your UI Changes
Everything above this line is the easy half, and it was always the easy half. Three jobs, a job dependency, three exit codes: none of that changes once it's written. What breaks a smoke gate in practice is not the CI wiring. It's the suite it calls.
A hand-maintained smoke suite has always been a curated list a human wrote once: four checks, ten checks, whatever the product needed the day someone decided. That list decays at exactly the rate the product changes shape. A selector moves, a route gets renamed, a new checkout step gets inserted between cart and payment, and the suite either breaks on a UI change it was never testing in the first place, or worse, keeps passing against a flow that no longer represents what a user does. Someone has to notice, open the spec file, and fix it by hand, on top of whatever else shipped that week. That's the maintenance the CI wiring was never solving.
We built Autonoma around the idea that the suite itself shouldn't be something a person maintains at all. Our agents read the codebase directly, the routes, the components, the flows that actually exist, and generate checks against them, then run those checks against the live, deployed candidate instead of a saved snapshot of the DOM. On every pull request, the Diffs Agent re-reads what the diff changed and updates the suite to match: a renamed button gets picked up, a route that no longer exists gets dropped, a new critical path gets added, without anyone opening the spec file by hand. The suite stops being a list someone maintains and becomes something regenerated against whatever the codebase looks like this week.
Mapped onto the workflow above, that changes what the smoke script calls, not the contract it honors. The exit codes stay 0, 1, and 2. The job dependency still blocks a bad promote. What changes is who keeps the suite behind that call accurate as the product moves.
The Maintenance Math: Why the CI Wiring Was Never the Hard Part
Wiring three jobs into a workflow file is a few hours of work, done once. That was never the part that made teams disable their smoke gate six months in.
What makes a gate get disabled is the slow accumulation of false results: a check that fails because a class name changed, not because anything actually broke, repeated often enough that someone quietly lets the step continue on error just to stop the noise, which turns a gate back into the advisory check the previous section described. The suite didn't get worse on purpose. It just stopped matching the product faster than anyone had time to fix it, and the fastest way to make a broken alarm stop going off is to disconnect it.
Both suites start identical. They diverge the first time the product changes shape.
A hand-maintained suite and a codebase-derived one start from the same four checks and the same workflow file. They diverge the first time the product changes shape underneath them, which for most teams is the following sprint.
What the Smoke Gate Should Not Be Asked to Do
A blocking gate is a strong tool used on a narrow job, and stretching it past that job is how teams end up trusting a green smoke run for more than it's telling them. It answers one question, is this candidate alive enough to promote, and it answers it in minutes because it never looks past one level of depth.
Some kinds of testing belong nowhere near this gate, on purpose. A beta program that puts a build in front of real users is a judgment call about product fit, not a pass or fail check. Exploratory testing, someone poking at the product because something felt off, needs a human's instinct, not a fixed assertion. Load testing measures concurrent traffic the smoke suite's four checks never generate. Accessibility testing needs a scanner, or a person, built for that job specifically. Unit-level structural coverage belongs to the runner closest to the code, not a suite that drives a browser against a deployed candidate. Contract testing between two services checks an interface agreement this suite was never designed to verify.
None of that is a gap in this gate. It's the boundary of what a blocking pre-promote check should be responsible for, and naming it honestly is part of building one that works. The CI/CD testing guide covers what belongs around this workflow, and API smoke testing covers the service-layer equivalent of this browser-level gate; how testing threads through delivery is worth reading next if the question is bigger than this one gate.
If a smoke suite already exists somewhere on your team, in a spec file, in a wiki page describing ten manual steps, in someone's memory of what to check before a release, the workflow above is what turns it into something that actually blocks. If the suite doesn't exist yet, or nobody trusts it to stay accurate past the next redesign, connecting the codebase to Autonoma is the faster path to the same blocking gate, minus the part where someone rewrites the spec file every time the UI moves. Either way, the contract above, three exit codes, one blocking dependency, doesn't change.
Frequently Asked Questions
Structure the workflow as three jobs: a build job that compiles the app and deploys a preview candidate, a smoke job that depends on the build job and runs the suite against that candidate, and a promote job that depends on the smoke job. Because the promote job's dependency includes the smoke job, GitHub Actions will not run it unless the smoke job reports success. The suite itself can be any test runner; the gate is created by the job dependency, not by which framework runs the checks.
It should block. A smoke check that posts a result to a channel without stopping the promote step is an advisory check, not a gate, and it produces the same outcome as having no check at all: the build ships regardless of what the suite found. The only thing that makes a check a gate is a structural dependency where a red result prevents the next job from running.
Distinguish a real check failure from an infrastructure problem. A common contract is exit 0 for all checks passing, exit 1 when the suite ran successfully and a real check failed, and a distinct code such as exit 2 when the harness itself failed to run, for example the preview environment never came up or the browser failed to launch. Collapsing both failure types into one code makes every flaky runner look like a product bug.
No. CI/CD testing is the broader discipline of testing at every stage of a delivery pipeline, from unit tests through integration checks to release gates. Smoke testing is one specific gate inside that pipeline, positioned after the build and before promotion, and it exists alongside unit runners, regression suites, and other checks rather than replacing any of them.
It does not need to, and that is what makes it easy to adopt: the workflow, the job dependency, and the exit-code contract stay exactly as described here, so there is no pipeline rework to do. What Autonoma changes is the suite behind the gate. Our agents read the codebase to generate the checks and update them automatically as the UI changes, so the workflow keeps calling a suite that still matches the product instead of one a person has to rewrite by hand. You keep the gate you already built and stop maintaining the tests it runs.
Judgment-based checks that a script cannot evaluate: beta programs that gauge real user reaction, exploratory testing where a person probes for anything that feels off, and decisions about whether a product change is a good idea in the first place. A smoke gate answers whether a build is alive enough to promote. It was never built to answer whether the build is a good idea.
Any runner that returns a meaningful exit code can serve as one. The gate is created by the job dependency and the exit-code contract, not by which framework runs the checks, which is why the workflow above stays runner-agnostic: Playwright, Cypress, a shell script hitting a health endpoint, or something else entirely all plug into the same three-job structure.
