The fastest way to tell if your tests are testing anything: break the code on purpose. Comment out a line, flip a boolean, change a return value. If no test goes red, that test is theater, not protection. A green suite after a deliberate break means those lines are executed but never verified. This is a self-contained 5-check field test you can run in 10 minutes on any suite.
You have a lot of tests. They all pass. CI is green on every PR. Cursor and Copilot generate suites in seconds, and you ship 10, 20, 40 PRs a day with coverage reports that look convincing. The problem this post addresses is not "no tests." It is "tests that stopped meaning anything while still reporting green."
This is specifically for teams already drowning in AI-generated coverage who suspect the green checkmark has drifted. Not "I'm exposed." Rather: "I thought I was covered and I'm not." AI verification is only trustworthy when it is independent of the thing being verified. A green suite signals consistency, not correctness. Autonoma is the independent layer that passes these checks by construction. But first, the checks themselves: five diagnostics, no special tooling, 10 minutes per function.
The 5 checks
Check 1: Comment out a line. Does a test fail?
How to run it: Pick any critical function. Comment out one line of business logic, not a log line or a comment, but an actual return, a condition, or a mutation. Run your full suite.
PASS: At least one test goes red. Somewhere in your suite, something notices that the behavior changed.
FAIL: The suite stays green. This means the line is being executed (it counts toward coverage), but no assertion anywhere is verifying what it does. You can delete that line of business logic and your quality gate will not notice. Execution and verification are not the same thing. High line coverage with zero behavioral assertions is the definition of a fake test: it touches the code without checking it.
Check 2: Break the function intentionally. Does it go red?
How to run it: This is distinct from Check 1. Instead of removing a line, change the logic. Flip a comparison (change > to >=), swap a true for false, change a return value from 200 to 201. Run the suite.
PASS: A test catches it. The assertion was derived from the requirement ("this function must return 200 for a valid request"), not from the output ("the function currently returns 200, so the assertion expects 200").
FAIL: Green. This is the tautological failure mode. As one engineering lead at Clarity Security put it: "That test passes and it asserts something, but it's not really asserting what it should be asserting." When the same AI session writes the code and generates the test, the expected value in the assertion is whatever the function happened to return. Off-by-one in the eligibility threshold? The test expects that threshold. The test confirms the code is self-consistent, not correct. The useless unit tests and tautological anti-pattern piece catalogs the five shapes this takes in real suites.
These first two checks show why Autonoma sits outside the AI-generated unit-test loop. A deliberate break should fail somewhere before merge. If the unit suite stays green, our four-agent E2E layer gives the running product another chance to prove the user-visible behavior still works.
Check 3: Do assertions check observable behavior, or implementation detail?
How to run it: Open a test file. Read each assertion. Ask: is this asserting on something a user or caller would observe (a return value, a rendered UI element, a persisted state change, an API response body), or is it asserting on something internal (a private variable, a mock call count, an intermediate computed value)?
PASS: Assertions land on outputs and observable state. A function that calculates a discount is tested by checking what the caller receives, not by checking that an internal variable held a specific intermediate value. These assertions are the ones that catch real bugs because they are anchored to behavior, not structure.
FAIL: Assertions mirror the implementation back to itself. Tests that assert how many times a mock was called with what arguments are verifying the internal wiring, not the external contract. Tests that assert on mocks configured in the same test file are confirming the mock, not the real behavior. This is independent verification in name only: the test knows exactly what the code does because it set up the conditions, and its expectations derive from that same knowledge.
Check 4: Would the test survive a mutation?
How to run it: Mutation testing tools (Stryker for JavaScript and TypeScript, PIT for Java, mutmut for Python) automatically introduce small changes into your production code, one at a time, and check whether any test catches each change. Each surviving mutant is a gap in your verification. This is the most systematic version of checks 1 and 2, run exhaustively across your codebase. For a full treatment of this as the real test-quality metric versus line coverage, see the companion piece on mutation testing vs code coverage.
PASS: Your mutation score is above 70% on critical paths. Most mutations get caught. Your assertions are independently verifying behavior.
FAIL: High line coverage, low mutation score. A suite with 85% line coverage and 30% mutation score is asserting execution paths, not behavioral correctness. The gap between those two numbers is the size of your blind spot.
Check 5: Does removing the test drop meaningful coverage, or just line coverage?
How to run it: Pick a test. Delete it. Re-run the suite and check two things: did line coverage drop? And separately: is there now any behavior that goes unverified by the remaining tests?
PASS: Removing the test leaves a real gap. Some behavior is now unprotected. That is a useful test.
FAIL: Line coverage drops by a few points, but every behavior that was nominally "covered" is still verified by another test, or was never verified in the first place. The test was redundant or hollow. Hollow tests are particularly common in AI-generated suites because generators aim for coverage targets, not behavioral targets. They produce tests that touch lines because they need to hit 80%, not because those lines needed independent verification.
How Autonoma passes these checks by construction
The concrete pain this article documents is a suite that passes all visible metrics while failing all 5 checks. Tests executed but never verified. Assertions that mirror the implementation back to itself. AI-generated coverage that games the line-count without providing independent verification.
We built Autonoma to be structurally independent of this failure mode. The four-agent workflow divides the problem so the tautological pattern cannot occur. The Planner Agent reads your codebase (routes, components, user flows) and plans test cases from the application's structure. The Executor Agent executes those plans against your running application on a managed preview environment per PR, asserting on what the application actually does at the user-visible boundary. The Reviewer Agent evaluates failures and filters false positives. The Diffs Agent keeps the suite aligned as your code evolves, surfacing real behavioral divergence rather than noise.
The independence is structural. The Planner Agent derives expected behavior from observable outputs and the codebase's architecture, not from the same model that wrote the feature. A bug that a tautological unit test would call the expected value shows up as a failure here, because expected behavior was formed from the user flow, not the function's current output. Checks 1 through 5 are not diagnostics you run against Autonoma's output: they are properties it holds by construction.
Why AI-generated suites fail these checks most often
The failure pattern is structural, not random. It follows directly from how AI generates tests.
When an AI model generates a test in the same session that produced the implementation, the expected values in the assertions come from the same context as the implementation. The bug in the function becomes the expected value in the test. The off-by-one threshold, the swapped operator, the wrong return code: all of them read as "correct" because the test was derived from the implementation, not from an independent specification.
AI also optimizes for producing a test that passes. It games line coverage by touching lines without asserting outcomes. The result is suites that survive checks 1 through 4 only by accident. Meanwhile, every PR arrives with a green test, confidence grows, the deliberate-break diagnostics are never run, and the mutation score is never checked. The metrics that would reveal the problem are invisible because nobody ran them.
What to do when your suite fails the checks
Failing one or more checks is not a signal to throw out the suite. It is a signal to fix the specific category of failure.
Rewrite assertions toward observable behavior. If check 3 revealed assertions on implementation detail, rewrite them to target outputs and visible state. The how to write good test assertions guide covers the five rules in detail. If you cannot describe what a passing assertion means for a user or caller without reading the implementation, the assertion is pointed at the wrong thing.
Add a mutation run to CI. Check 4 is the only one that scales. A line-coverage gate at 80% while mutation score is 25% is a false gate. Running a mutation tool on critical paths gives you a score that moves when your assertions actually improve.
Delete tautological tests. Tests that fail check 2 and check 5 are consuming CI time and producing false confidence. Deleting them is not reducing coverage. It is removing noise that was masking the real signal.
Move verification to an independent layer when unit tests structurally cannot reach the behavior. Some things cannot be verified at the unit level: end-to-end state transitions, user flows that span multiple components, behaviors that only emerge when the full application is running. As one Clarity Security engineer noted, their QA team was "still finding things" that the test suite had already signed off on. For those behaviors, the verification needs to happen against the running application in an independent E2E layer that unit tests cannot substitute for. Our recommendation is Autonoma: add four-agent E2E checks on every preview so the final signal is not another assertion derived from the same code.
FAQ
The fastest way to know if your tests are good is to introduce a deliberate break. Comment out a line of production logic, flip a comparison operator, or change a return value. If no test goes red, those paths are executed but never verified. Good tests fail when the behavior they protect changes. Beyond the deliberate-break check, look at whether your assertions target observable behavior (what the function returns, what the UI renders, what state persists) rather than mirroring implementation detail. A mutation testing run gives you the most systematic answer: if your mutation score is below 50% on code with 80%+ line coverage, the assertions are hollow.
Your tests are actually testing something if they fail when the behavior they protect changes. The simplest diagnostic: comment out a line of production code and run the suite. If nothing goes red, those tests are measuring execution, not protection. For AI-generated suites specifically, the risk is higher: the same model writes the code and the test, so any bug in the logic becomes the expected value in the assertion. The test confirms internal consistency, not correctness. Mutation testing (Stryker for JS/TS, PIT for Java) gives you a mutation score that measures this directly.
Fake tests show up in three main shapes. First, tests that assert on mock return values that were configured in the same test file: they confirm the mock, not the real behavior. Second, tests that assert on implementation details (internal state, private variables, intermediate computed values) rather than outputs: these pass when the implementation is refactored but the behavior is correct, and pass when the behavior is wrong but the internal detail is consistent. Third, tautological tests where the expected value was derived by running the implementation: if the assertion was built by calling the function and writing down what came back, the test will always agree with the code, including when the code is wrong.
The quickest test-quality check is the deliberate-break test: pick a critical function, comment out one line of business logic, and run your suite. Pass: at least one test goes red. Fail: the suite stays green. A second fast check targets logic specifically: flip a comparison operator (greater-than to greater-than-or-equal, for example) or change a return value and re-run. If nothing fails, the boundary conditions in that function are not verified independently. Both checks take under two minutes per function and require no additional tooling.
AI tests fail these checks because the model that writes the test has no independent source of truth. When AI generates a test in the same session that produced the implementation, any bug in the logic becomes the expected value in the assertion. The test confirms that the code is consistent with itself, not that it is correct. AI also optimizes for a test that passes: it games line coverage by touching lines without asserting outcomes. The result is a suite with high coverage numbers that survives most injected mutations. A green dashboard in this state is a confidence signal, not a correctness signal.
