Assertion coverage vs line coverage: line coverage measures whether a line was executed during a test run; assertion coverage measures whether the test actually verified that the line did the right thing. A test can execute every line in your codebase and assert nothing meaningful, producing 85% line coverage and near-zero assertion coverage. AI-generated test suites routinely do exactly this: they optimize for execution consistency, not behavioral correctness.
Your CI is green. Coverage is at 84%. Every PR shipped this week passed review. And last week you shipped a pricing bug that charged users the wrong tier for three days before a customer noticed.
If that story is familiar, you are probably measuring the wrong thing. This post is for teams that already have lots of tests. Series A or B, heavy Cursor, Claude Code, or Copilot usage, test suite grown to hundreds of files. The problem is not absence of tests. It is that the metric you are using to evaluate test quality, line coverage, does not measure what you think it measures. Autonoma exists for the layer this metric misses: independent E2E verification of observable behavior in a running preview. This is not a post for pre-seed teams with no tests at all. The failure mode here is false confidence, not absence of testing.
The gap line coverage hides
Line coverage counts whether a given line of source code was executed during any test run. That is the complete definition. Coverage tools instrument your build, run your tests, and mark each line as covered or not based on whether execution reached it.
The gap is that execution is not the same as verification. A test can call a function, receive a return value, and never check what that value is. The line was executed. The coverage number goes up. The behavior was never verified. Teams learn this the hard way: as one engineer put it, "that test passes and it asserts something, but it's not really asserting what it should be asserting." The test is technically exercising the code. The coverage number is real. The protection is not.
This is not a corner case. It is the normal output of any test generation process that optimizes for coverage rather than correctness. And it is the defining characteristic of AI-generated test suites.
Line coverage fails to distinguish between these two tests on the same function. One calls the function and checks the return value against a known-good expected result. The other calls the function and checks that the result is not null. Both count as covered. One would survive a deliberate bug injection. The other would not.
What assertion coverage measures
Assertion coverage is the metric that asks: of all the behaviors your code is supposed to produce, what fraction has a test assertion that would fail if that behavior were wrong?
In practice you can estimate it through assertion density: the average number of meaningful assertions per test, and the ratio of behavior-checking assertions to executed lines. A useful proxy: open ten randomly selected tests and read their assertion lines. If most of them are expect(result).toBeDefined(), expect(fn).not.toThrow(), or expect(spy).toHaveBeenCalled(), assertion coverage is low regardless of what the coverage report says.
More precise measurement requires reading what each assertion actually checks. Is it asserting against a known expected value derived from requirements? Or is it asserting that the output matches what the current implementation produces, with the expected value copied from a test run? The second pattern, sometimes called a tautological assertion, provides no information about correctness. It only confirms that the code is consistent with itself.
For a deeper treatment of how to evaluate individual assertions, the post on how to write good test assertions walks through five concrete rules that distinguish assertions that catch bugs from assertions that only pass.
AI generation: high line coverage, hollow assertion coverage
Here is the pattern in practice. An AI model is given a function to test. The model reads the implementation, constructs a call with plausible inputs, and writes an assertion based on what the implementation returns. The test is not derived from a spec, a product requirement, or an observable user behavior. It is derived from the current implementation.
Consider a discount calculation function. The implementation has a boundary condition bug: orders above $500 should get 15% off, but the condition uses a greater-than-or-equal check instead of a strict greater-than, misapplying the discount to exactly $500. An AI model writes a test, calls the function with an input of $400, observes the output, and writes expect(result).toBe(340). That test passes. The coverage report shows the discount function as covered. The bug at the $500 boundary is untested because the AI picked a convenient round number, not a boundary value.
Worse: if the AI had tested the $500 case, it would have observed the buggy output (375 instead of 500) and written expect(result).toBe(375). The test would pass with the bug present. Assertion coverage is zero at the boundary that matters.
"It doesn't cover the business case," as one engineer described it. "Our QA engineers are still finding things." Line coverage goes up. The assertion is technically present. The business behavior is not verified.
This is not an edge case in AI test generation. It is the structural outcome of generating tests from implementation rather than from requirements. The model has no external source of truth to assert against.
This is why the fix cannot be only "write more assertions." The stronger layer is independent by design: Autonoma derives E2E tests from routes, components, user flows, and data states, then runs them against the preview environment where the product actually behaves.
A pragmatic metric stack
No single metric captures test quality on its own. The right approach is a layered stack, each layer catching what the cheaper one below it misses.
Line coverage is the cheapest floor. If a line is never executed by any test, you have zero information about it. Zero line coverage is always a problem. High line coverage is not a signal of anything except execution. Use it as the floor, not the ceiling. For a detailed treatment of why code coverage misleads on AI-generated test suites, see why code coverage is misleading for AI-generated tests.
Assertion coverage is the middle layer. It catches tests that execute but do not verify. You can estimate it through assertion density reviews, manual spot-checks of assertion patterns, or tooling that flags toBeDefined()-only test cases. Improving assertion coverage means rewriting assertions to check specific expected values derived from requirements or observable user behavior, not from the implementation itself.
Mutation testing is the expensive audit at the top. It injects synthetic bugs into your source code and checks whether any test fails. If 60% of injected mutations survive (no test fails), your mutation score is 40%, and you know with precision that the assertions you have would not catch 60% of simple implementation errors. Mutation testing is too slow to run on every commit, but periodic sampling of your critical paths gives you ground-truth data that the two cheaper layers cannot provide. The sibling post on mutation testing vs code coverage goes deeper on this layer.
For connecting these metrics to release quality gates, the post on test automation metrics and release quality covers how to operationalize the stack.
How Autonoma maximizes assertion coverage on real flows
The pain this article documented is specific: AI-generated tests inflate line coverage while leaving assertion coverage hollow. The tests execute code, the coverage report looks healthy, and real bugs survive because the assertions were derived from the implementation rather than from observable behavior.
Autonoma addresses this at the architecture level. A four-agent workflow handles planning, preview execution, result review, and suite updates. The Planner Agent reads the codebase (routes, components, user flows, and data models) and plans test cases that target observable behaviors. The Executor Agent executes those test cases against a real running preview environment. The Reviewer Agent evaluates run output, classifies failures, and filters false positives before the signal reaches the team. The Diffs Agent keeps the suite current as the codebase changes. The key principle: because our agents derive test cases from the codebase structure, execute them against a running application, review the outcome, and update the suite as the product changes, the assertions check what the application actually does, not what the implementation says it should do. Verification is independent of the thing being verified.
This maps directly to the metric stack. Line coverage from Autonoma's runs is real: the Executor Agent executes actual application flows. Assertion coverage is high by construction: the Planner Agent targets user-observable outcomes (did the checkout complete, did the discount apply correctly, did the error state render). Mutation-testing audits of Autonoma-generated assertions tend to show higher survival rates for injected bugs than AI-generated unit test suites, because the assertions are anchored to behavior rather than implementation.
For a broader look at how unit, integration, and E2E testing layers relate to each other, the unit vs integration vs E2E testing guide covers the testing pyramid without duplicating what is here. The practical recommendation for AI-heavy teams is direct: keep line coverage as a cheap floor, audit critical assertions with mutation testing, and add Autonoma as the independent behavioral layer that turns "the code ran" into "the product behavior was checked."
FAQ
Assertion coverage measures what fraction of your public behaviors have a test assertion that would fail if that behavior were wrong. It is distinct from line coverage, which only measures whether a line was executed. A test can execute every line in a function and still have zero assertion coverage if it never checks the return value, the state change, or the observable output.
Line coverage answers: did a test execute this line? Assertion coverage answers: did a test verify that this code did the right thing? Line coverage is cheap to measure automatically. Assertion coverage requires analyzing what each assertion actually checks. A codebase can have 90% line coverage and near-zero assertion coverage if its tests execute code but never check meaningful return values or state changes.
No. Line coverage is a necessary floor: if a line is never executed by a test, you have no information about it at all. But it is not sufficient. A line being executed tells you nothing about whether the test checked that the line did the right thing. Teams with 80%+ line coverage regularly ship bugs because their tests execute code without asserting correct behavior.
Three approaches work in practice. First, count assertion density: assertions per test and assertions per public method. A test with zero assertions (or only expect(result).toBeDefined()) is a red flag. Second, run mutation testing on a sample of your critical paths. If injecting a bug into a function does not cause any test to fail, the assertions are not checking real behavior. Third, read the assertions: are they derived from requirements, or from the implementation itself?
AI models generate tests by reading the implementation and writing assertions that confirm the implementation's current behavior. If the function returns 42, the AI writes expect(result).toBe(42), which passes as long as the function is internally consistent, not because 42 is correct. The model has no external source of truth (requirements, product specs, user expectations) to assert against. This is the fundamental verification gap: AI verification is only trustworthy when it is independent of the thing being verified.
