Testing streaming AI responses means verifying the response as a lifecycle, not a value: assert on partial-chunk validity, monotonic buffer growth, clean stream completion within a timeout, graceful recovery when the connection drops mid-stream, correct typing-indicator transitions, and the final assembled output against a semantic invariant rather than an exact string, since chunk boundaries and generated wording are both non-deterministic between runs.
The assertion passed on your laptop. In CI, it hung for thirty seconds and timed out. Nothing about the code changed, the difference was timing: locally the response landed before the test checked anything, in CI it hadn't, and the test had no idea what to do with a response only half in flight.
That's the tell. If your streaming AI feature has a test suite, it's almost certainly testing the finished string, not what a user experiences: a message that grows on screen for several seconds, one token at a time, with a UI making decisions about indicators and error states the whole time.
One team had a typing indicator reported stuck for forty seconds, on a response that finished streaming in under two. The text sat complete in the DOM while the indicator kept spinning underneath it. Nobody caught it, because every existing test waited for the full response and asserted on that. The bug lived entirely in the part no test was watching.
Why a Streamed Response Breaks Conventional Assertions
A conventional assertion assumes the thing you're checking already exists in full by the time you check it. Call an endpoint, get a response body, assert on it. That model breaks immediately against a streaming feature: at the moment your test fires the request, the response doesn't exist yet as a value. It's a sequence of events over a lifecycle: a connection opens, tokens arrive one at a time over EventSource or a WebSocket, the UI repaints on every chunk, and only later does a finished response exist to assert against.
Treat it like a normal HTTP call and you get one of two broken tests. Wait for the whole thing and assert on the final string, and you throw away every bug in the middle: the stuck indicator, the malformed chunk, the connection that drops at 60 percent. Or assert too early, and your test becomes a coin flip on network timing, flaking in CI for reasons unrelated to your code. If you've already worked through testing a chatbot, this is the same non-determinism problem one layer lower, down at the transport, not just the wording.
There's a reason evaluation tooling is no help here either. Evals score the finished response, which is precisely the artifact that exists only after every interesting failure has already happened. A stuck spinner has no representation in the text a model produced. Neither does partial output that vanished on re-render, or a scroll anchor that fought the user for four seconds. Those live in the running application, and the only way to assert on them is to drive the application while it streams. That's the discipline Autonoma is built around, and it's worth keeping in view as you read the assertion targets below, because roughly half of them are invisible to anything that inspects a response body.
The Stream Lifecycle Is a State Machine, Not a Value
Name the states and the problem gets tractable. idle is before the user has asked anything. requested fires the moment the request goes out, and it's also when a typing indicator should appear. first token is the instant the first chunk lands, arguably the most important transition, since it's usually when the indicator should disappear. streaming covers every chunk after, where the buffer grows and the UI repaints. complete is reached when a terminator arrives, a literal data: [DONE] line or a finish_reason field. error branches off streaming at any point the connection drops or a malformed frame arrives, and it's the transition most streaming UIs get wrong.
Five states, one error branch. Every assertion target below maps to a transition or a state in this same machine.
That's the point: once you stop asserting on "the response" and start asserting on specific transitions, flaky streaming tests mostly stop being flaky. They fail, or pass, for identifiable reasons tied to a specific state, not vague timing nobody can reproduce.
Partial-Response Correctness: Every Chunk Must Parse, and the Buffer Must Only Grow
Every chunk must be valid and safe to render on its own, and the buffer it produces must never be shorter than the buffer before it.
Most teams skip this because it feels like transport, not the AI feature. It doesn't skip cleanly: a malformed delta, an unescaped fragment inside a chat bubble, or a chunk that silently replaces the buffer instead of appending can render broken UI or rewrite a response mid-stream, invisible against the finished string. The fix: a parse check on every delta plus a running length check, the buffer's length after each chunk must be greater than or equal to its length before.
Stream Completion: The Terminator Must Arrive Inside a Timeout
The stream must reach its terminator, a literal data: [DONE] line or a finish_reason field, within a bounded time, and the client must leave the streaming state once it does.
A stream that never terminates isn't paused, it's broken, and the UI needs to know within a timeout short enough that a user isn't staring at a spinner for a full minute. Assert elapsed time against a fixed threshold on every run, not by confirming once in a debugger that the terminator eventually showed up.
Error Mid-Stream: The UI Must Recover, Not Freeze
When the connection drops or errors partway through, the client must surface the failure, keep whatever partial text already arrived, clear any loading state, and offer a way to retry.
This is the failure mode a happy-path test never exercises, because it never cuts the connection. Deliberately abort the stream partway (an abort switch, or a fulfilled request that stops after a few chunks) and check three things independently: partial text stays on screen, the indicator clears instead of spinning forever, and a retry control appears. Miss any one and you get a frequently reported bug: a UI that looks broken even though the network genuinely failed, and there was nothing the app could do about the failure, only about how it responded.
Same drop, two outcomes. The only difference between the two branches is what the client code does after the connection dies, not the failure itself.
The Typing Indicator: On at Request, Off at First Token, Never Stuck
The indicator must appear the moment a request goes out, disappear the moment the first token (or, if you choose that convention instead, the completion event) arrives, and never remain visible after an error.
State your convention explicitly, because "off at first token" and "off at completion" are both defensible, and testing the wrong one produces a false failure. Whichever you pick, the indicator's state is cheap to assert against real DOM, and it's the single most common bug in every streaming chat UI, not because it's hard to build correctly, but because it's easy to forget to clear on the one code path, the error path, nobody tests.
Note the phrase "against real DOM," because it's doing all the work in that sentence. There is no protocol-level assertion for a stuck indicator. The stream behaved perfectly; a component didn't unsubscribe. That is the shape of nearly every bug users actually report about streaming features, and it's why the behavioral layer isn't a nice-to-have here the way it might be for a CRUD form. Autonoma sits at exactly that layer, running end-to-end tests against the deployed application on the pull request, asserting on what the interface is doing mid-stream rather than on what eventually arrived.
Final Assembled Output: Assert Intent, Not Exact Text
Once every chunk lands, the concatenated result must satisfy what the user asked for, checked against invariants and semantic similarity rather than an exact string match.
Chunk boundaries are non-deterministic between runs, and if you're streaming from a real model the wording is too, so an assertion pinned to exact text will flake on a paraphrase that's completely correct. Run the same prompt several times and assert on what has to be true of the assembled buffer: it contains the fact you asked for, ends on a complete sentence, stays under a length bound. Never assert on chunk content or chunk count, both are accidents of network timing, not properties of the response. A streaming response is one instance of a wider pattern, a UI whose structure isn't fixed at request time, the same territory testing generative UI covers once the model generates layout, not just words.
Two Levels of Streaming Tests: A Fake Server and a Real Browser
Level 1 tests the stream protocol against a deterministic fake server: scripted tokens, no live model call, sub-second runs, safe on every commit. A fixture you control can be told to omit its terminator or die at chunk six on command, which a real model API won't reliably do.
Level 2 tests the streaming UI behaviorally, in a real browser, against the actual feature. This catches what Level 1 structurally cannot: an indicator that never clears because an event listener got detached, partial text that vanishes when a component re-renders on error, a scroll anchor that jumps mid-stream. None of those bugs live in the protocol. They live in the running application, so a fake-server unit test will never see them.
Most teams do Level 1 and stop, or skip both and learn about the stuck indicator from a support ticket. Level 1 is the easy half to build and the easy half to keep: it's hermetic, fast, and its fixture changes only when the protocol does. Level 2 is where the effort actually goes, and where suites tend to rot, because the assertions are tied to a real interface that gets redesigned while the streaming logic underneath stays put.
This is the split Autonoma is built to absorb. It runs behavioral end-to-end tests against the real deployed application in a preview environment on the pull request, so the Level 2 checks above, indicator on at request, off at first token, cleared after an error, partial text retained, run against the actual UI on every change rather than on whichever sprint someone had time. The Diffs Agent updates the suite from each pull request's diff, which matters for a streaming feature specifically: swapping a chat component or moving the indicator into a different subtree is a one-line diff that silently invalidates every hand-written selector pointed at it.
Keep Level 1 in your own repo regardless. Protocol invariants are yours, they're cheap, and no external tool should be your first line of defense against a malformed chunk. The division that works is protocol tests you own and maintain, behavioral tests that maintain themselves against the running app.
A Runnable Stream Test
Here's the fixture that makes this testable: a small FastAPI server streaming a scripted token sequence, with a param to cut the stream short and one to withhold the terminator on command. It also serves a minimal HTML page wired to the same stream, so one fixture backs both the tests below and the browser test after them.
Point a pytest suite at it and the invariants above become assertions instead of hopes: every chunk parses, the buffer only grows, the terminator arrives inside a timeout, and the assembled result is checked with a semantic, run-it-five-times assertion instead of an exact string.
Then force the failure mode a happy-path test never exercises: cut the connection mid-stream and check the client doesn't fabricate a terminator it never received, and retains whatever partial text arrived before the drop.
The Level 2 Playwright Test
The fixture's HTML page is deliberately minimal: an Ask button, a response element, a typing indicator, and a retry button that appears only after an error. That's enough for a real-browser test to check what the fake-server tests cannot: indicator state lines up with the lifecycle, response text grows over time instead of appearing all at once, and a request Playwright fulfills with a truncated body leaves partial text on screen, clears the indicator, and shows retry.
Wiring This Into CI Without Flaking
Run the fake-server suite on every commit, not on a schedule. It has no live model call and no network dependency beyond localhost, so there's no excuse to skip it. A job-level timeout of a few minutes catches the failure mode CI is actually vulnerable to: a stream that hangs instead of erroring cleanly, blocking the pipeline instead of failing it. Give the server a moment to boot, keep per-test timeouts short, and treat any test that needs a retry to pass as a bug in the assertion, not a reason to add retries.
That last point matters more for streaming than most other tests: a retry silently papers over the exact race, between buffer state, indicator state, and network state, that streaming introduces. Fix the wait condition. Don't add a retry.
Two things are worth carrying out of this piece. The first is that a streaming response is a lifecycle, and every assertion above is anchored to a named state or transition in it, which is what turns a category of test with a reputation for flaking into one that fails for reasons you can name. The second is that the lifecycle only half exists on the wire. The other half exists in the interface, in indicator state, retained partial text, and a retry control that either appeared or didn't, and no amount of protocol coverage reaches it.
Build the fake-server suite yourself and run it on every commit. Then put a behavioral layer under the interface, whether that's the Playwright spec above maintained by hand or Autonoma running it against your deployed app on each pull request. The stuck-indicator bug from the opening of this piece was never a model problem or a network problem. It was a UI that nothing was watching while it streamed.
Frequently Asked Questions
Test the lifecycle, not the finished string: assert that every chunk parses and the buffer only grows, that the stream reaches its terminator inside a timeout, that a mid-stream connection drop clears the loading state without wiping partial text, that the typing indicator's on and off transitions match the lifecycle, and that the assembled output satisfies a semantic or invariant check rather than an exact match.
Run a deterministic fake SSE server behind your tests so you can force every failure mode on demand: a normal completion, a stream that never sends its terminator, and a connection that drops at an arbitrary chunk. Assert against that fixture in a fast, hermetic pytest or vitest suite, then add a smaller set of real-browser tests against the actual UI to catch bugs, like a stuck indicator or wiped partial text, that only exist once a real component is rendering the stream.
Almost always because the assertion is checking timing instead of a lifecycle transition, or because a retry was added to paper over a race between buffer state, indicator state, and network state that a hard timeout and a correctly ordered assertion would have caught cleanly. Fix the wait condition, don't retry the test.
Never assert on exact text or on individual chunk content, since chunk boundaries and, against a real model, the wording itself are both non-deterministic between runs. Assert on invariants instead: the assembled buffer contains the required fact, ends on a complete sentence, stays under a length bound, and passes a semantic similarity check run across several repetitions of the same prompt rather than once.
Drive the real UI in a browser test and assert on DOM state at each lifecycle transition: visible the moment a request goes out, hidden the moment the first token (or completion, depending on which convention you picked) arrives, and, critically, hidden after an error too, since a stuck-forever indicator on the error path is the most commonly reported bug in streaming chat interfaces.
Because an eval scores the finished response, and a stuck indicator has no representation in the text the model produced. The stream can behave perfectly at the protocol level, with every chunk valid and the terminator arriving inside its timeout, while a component simply failed to unsubscribe on the error path. The same is true of partial text that vanished when a component re-rendered, and of a retry control that never appeared. Those exist only in the running interface, so catching them means driving the live UI while it streams. That's the layer Autonoma covers, running behavioral end-to-end tests against the deployed application rather than reading a transcript of what the model said.
