How to Test Non-Deterministic AI Outputs

Your CI run goes red. You rerun the job. It goes green. You ship. Two days later the exact same test fails on the exact same input, and this time nobody reruns it, because it's 4pm on a Friday and the failure looks like every other flaky test that's ever cried wolf in that pipeline.

That flake didn't come from a network blip or a race condition in your test runner. It came from the model. Somewhere in your suite, an assertion expects the exact string a language model returned the last time this test happened to run, and the fix everyone reaches for first, retrying until it goes green, is exactly backwards. Retrying a network timeout recovers a false negative. Retrying a non-deterministic AI test just hides the one signal you actually have: that the same input can produce a different, and sometimes wrong, output.

Why Does My Chatbot Give Different Answers to the Same Question?

Ask a language model the same question twice and you'll often get two different, both defensible, answers. Four things cause that, and only one of them lives in a prompt file you control.

Sampling temperature and top-p are the obvious ones: the model doesn't always pick the single highest-probability next token, it samples from a distribution, so a higher temperature means more wording variation from one call to the next. Context and retrieval drift is the second, and it's the one teams miss most often: if your feature pulls in a RAG chunk, a tool result, or recent conversation history, and that upstream data shifted even slightly between two calls, the model is answering a subtly different question each time even though the user typed identical words. Prompt phrasing is the third: a system prompt edited last sprint, one extra sentence a teammate added "for clarity," and the model's behavior shifts in ways nobody flagged as a change. The fourth sits entirely outside your codebase: providers update the underlying model silently, behind the same model name and the same API contract, so your exact prompt now hits a different set of weights than it did last week, with no diff for you to review.

Here's the correction that trips up almost everyone building on this stack: temperature=0 is not determinism.

Batched inference means your request shares a hardware batch with other requests in flight, and floating-point operations aren't strictly associative, so the same input can produce different logits depending on what else happens to be in that batch. GPU kernels introduce their own non-determinism at the level of floating-point precision. Mixture-of-experts models route tokens to different experts based on load, and that routing can vary run to run even when temperature is pinned at zero. This detailed teardown of batch invariance in LLM inference covers exactly why batch composition and floating-point non-associativity move outputs even when nothing about your request changed. And a silent provider-side model update changes output at temperature zero exactly as easily as it changes output at temperature 0.7. If your entire test strategy rests on temperature=0 producing a fixed string, it will pass in your local run and flake in production the first time any one of those four things moves, and you won't have a stack trace that tells you which one.

Testing Non-Deterministic Software: The Assertion Has to Change Shape

Deterministic software testing runs on an assumption so basic it's invisible: same input, same output, every time. You assert equality and move on with your day. Non-deterministic AI output breaks that assumption at the root, which means the fix isn't a looser number in the same kind of assertion, it's a different kind of assertion entirely.

The shift is this: instead of asserting that an output IS a particular value, you assert what must be TRUE about it. And instead of trusting a single run to represent the system, you trust a rate measured across several.

Every pattern below is one way of putting that shift into code. None of them require you to give up on assertions, or to accept that "AI is just fuzzy." They require you to assert the right thing.

Pattern One: Property-Based and Invariant Assertions

The most reliable fix for a flaky exact-match assertion is usually to stop asserting an exact match. Instead, assert the properties that must hold no matter which acceptable phrasing the model lands on this run: the response is well-formed and non-empty, it's under a length bound, it contains the entities the answer actually depends on (an order ID, a confirmation number, a status the user asked about), and it does not contain content you've explicitly forbidden (a leaked system prompt, an unqualified promise the business can't back).

This is deliberately the cheapest pattern in this article to write and the first one you should reach for. It needs no embeddings, no second model call, and no judge. If your response fails a structural invariant, you have a real bug, and chasing it into a semantic-similarity check first only adds latency to finding what's already broken.

Here's a pytest module asserting structural invariants against a stub model client, including the case where temperature=0 still has to hold across many draws, not just one:

"""
Property-based / invariant assertions for non-deterministic model output.

Instead of asserting the exact string a model returns, assert what must
always be true about it: valid structure, required entities present,
forbidden content absent, and bounds on length. These properties hold
across every acceptable phrasing, so they don't break when the wording
changes between runs.

Run with: pytest tests/test_invariants.py -v
"""
import re
import sys
import os

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from model_client import call_model  # noqa: E402

FORBIDDEN_PHRASES = [
    "as an ai language model",
    "i cannot access",
    "system prompt",
]

MAX_RESPONSE_CHARS = 400
ORDER_ID_PATTERN = re.compile(r"\b\d{3,6}\b")


def assert_invariants(response: str, expected_order_id: str) -> None:
    assert isinstance(response, str) and response.strip(), "response must be a non-empty string"
    assert len(response) <= MAX_RESPONSE_CHARS, f"response exceeded {MAX_RESPONSE_CHARS} chars: {len(response)}"

    lowered = response.lower()
    for phrase in FORBIDDEN_PHRASES:
        assert phrase not in lowered, f"forbidden phrase leaked into response: {phrase!r}"

    order_ids_found = ORDER_ID_PATTERN.findall(response)
    assert expected_order_id in order_ids_found, (
        f"expected order id {expected_order_id} not present in response: {response!r}"
    )


def test_refund_response_holds_invariants_across_runs():
    prompt = "What's the status of my refund for order 4821?"
    for _ in range(10):
        response = call_model(prompt, temperature=0.7)
        assert_invariants(response, expected_order_id="4821")


def test_refund_response_invariants_hold_even_at_temperature_zero():
    # temperature=0 reduces variance, it does not eliminate it. The
    # invariants must hold on every draw, not just the modal one.
    prompt = "What's the status of my refund for order 4821?"
    for seed in range(20):
        response = call_model(prompt, temperature=0, seed=seed)
        assert_invariants(response, expected_order_id="4821")

Notice what these assertions do not check: whether the wording is good, or whether the words match some reference answer. That's on purpose. Invariants are the floor every acceptable response has to clear, not the full definition of a correct one.

Pattern Two: Semantic Similarity Instead of Exact Match

Once your invariants pass, the remaining question is usually whether the free-text response means the right thing, not whether it's spelled the right way. Exact-match assertions can't answer that, because the same correct answer has effectively infinite valid phrasings. The standard fix is to embed both the response and a reference answer, compute cosine similarity between the two vectors, and assert the similarity clears a threshold.

The threshold is the part teams get wrong. Picking 0.8 because it sounds reasonable is a guess, not a measurement. The right way to set it: gather a sample of response pairs, label each pair "same meaning" or "different meaning" by hand, embed all of them, and pick the cutoff that separates your two labeled groups with the fewest misclassifications. That threshold is specific to your domain: a cutoff tuned on refund-status replies will not transfer cleanly to a domain where subtle wording differences change the actual claim being made, like medical or financial guidance.

Similarity also has a ceiling. It tells you two strings are close in meaning, not that either one is factually correct, or that a multi-step claim actually holds up. When correctness depends on more than paraphrase-level meaning, an LLM-as-judge scored against an explicit rubric is the right tool, not a tighter similarity threshold; that judge-based approach is exactly what our guide to prompt-level unit testing covers in depth.

Here's a semantic-similarity helper with the threshold calibration documented directly in the code:

"""
Semantic-similarity assertion helper.

Two correct responses can be worded completely differently. Comparing
them with string equality treats that as a failure; comparing them by
meaning does not. This module embeds two strings and asserts their
cosine similarity clears a threshold, calibrated empirically rather
than guessed.

Run with: python lib/semantic_similarity.py
"""
from sentence_transformers import SentenceTransformer
import numpy as np

_model = SentenceTransformer("all-MiniLM-L6-v2")

# This threshold was NOT picked arbitrarily. It came from embedding 40
# human-labeled pairs (20 labeled "same meaning", 20 labeled "different
# meaning") for this specific domain and picking the cutoff that
# separated the two groups with the fewest misclassifications. Re-run
# that calibration on your own labeled sample before trusting this
# number in a different domain; a threshold tuned for refund-status
# replies will not transfer cleanly to, say, medical triage responses.
#
# When correctness depends on more than paraphrase-level meaning (a
# multi-step claim, a fact that needs checking against a source), a
# similarity threshold isn't enough. That's when an LLM-as-judge scored
# against an explicit rubric earns its cost, not a tighter number here.
SIMILARITY_THRESHOLD = 0.82


def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))


def is_semantically_equivalent(response: str, reference: str, threshold: float = SIMILARITY_THRESHOLD) -> bool:
    embeddings = _model.encode([response, reference])
    similarity = cosine_similarity(embeddings[0], embeddings[1])
    return similarity >= threshold


def assert_semantically_equivalent(response: str, reference: str, threshold: float = SIMILARITY_THRESHOLD) -> None:
    embeddings = _model.encode([response, reference])
    similarity = cosine_similarity(embeddings[0], embeddings[1])
    assert similarity >= threshold, (
        f"semantic similarity {similarity:.3f} fell below threshold {threshold}: "
        f"{response!r} vs {reference!r}"
    )


if __name__ == "__main__":
    reference = "Refund processed for order 4821, 3-5 days"
    for candidate in [
        "We've completed the refund on order 4821. Expect it in your account within 3 to 5 business days.",
        "Sorry, I can't help with that.",
    ]:
        print(candidate, "->", is_semantically_equivalent(candidate, reference))

Passing a similarity check only tells you the words are equivalent, not that the right thing happened inside your application, which is exactly the gap behavioral end-to-end testing closes: Autonoma checks what an AI feature actually caused in a running app (the right screen, the right record, the right side effect) instead of the exact words it produced, so wording-level variance never enters into it.

How Autonoma Tests the Outcome Around a Non-Deterministic Response

Autonoma handles the product layer that a semantic score cannot see. Its Planner reads the routes, components, and flows in your codebase to plan behavioral tests for the running app, and its Executor drives the UI against a live preview environment to verify the state the response caused: the ticket moved, the refund amount changed, or the correct screen rendered. That keeps the assertion focused on a deterministic product outcome even when the model can phrase the response many valid ways. Diffs Agent then updates the affected coverage from the code diff, so a UI or flow change does not leave that behavioral check behind.

Pattern Three: N-Run Sampling and Self-Consistency

Testing the consistency of LLM outputs starts with accepting that a single run is never enough evidence for a non-deterministic system, no matter which assertion you're using. The fix is to run the same input multiple times, score each run independently, and report a pass rate instead of a verdict. This generalizes the self-consistency method introduced by Wang et al., from raising model accuracy to measuring test reliability. Two ways to turn that rate into a boolean: majority vote, where you require some fraction of runs (5 runs with 4 passing is a common default) rather than unanimity, and mean-similarity, where you average the similarity score across every run and threshold the mean instead of thresholding each run in isolation.

Choosing how many runs to sample is a cost tradeoff, not a purity contest. Five runs catches a scenario that fails more than roughly one time in five without paying for twenty-plus model calls on every assertion in your suite. Raise the count for scenarios you've watched flip near the majority line; a scenario that's a clean 5-of-5 or 0-of-5 rarely needs a bigger sample to trust.

Here's a sampling runner that executes the same input N times and reports both a majority-vote verdict and a mean-similarity verdict, instead of collapsing five distinct outcomes into one line:

"""
n-run sampling / self-consistency runner.

Non-deterministic output means a single run is never enough evidence.
This module runs the same input N times against a check function and
reports the pass rate, not just a single verdict, plus two ways to
turn that spread into a boolean: majority vote and mean-similarity.

Run with: python lib/sampling.py
"""
import os
import sys
from dataclasses import dataclass, field
from typing import Callable, List


@dataclass
class SamplingResult:
    n: int
    passes: int
    failures: List[str] = field(default_factory=list)

    @property
    def pass_rate(self) -> float:
        return self.passes / self.n if self.n else 0.0

    def majority_passed(self, majority: float = 0.6) -> bool:
        return self.pass_rate >= majority


def run_n_times(input_fn: Callable[[], str], check_fn: Callable[[str], bool], n: int = 5) -> SamplingResult:
    """
    Call `input_fn` n times (each call should independently invoke the
    model, so run-to-run variance is real, not memoized) and score each
    output with `check_fn`.

    Choosing n is a cost tradeoff: n=5 catches a scenario that fails
    more than roughly 1 in 5 times without paying for 20+ model calls
    per assertion. Raise n for scenarios you've seen flip results near
    the majority line; n=5 with a clear 5-of-5 or 0-of-5 result rarely
    needs a bigger sample.
    """
    result = SamplingResult(n=n, passes=0)
    for _ in range(n):
        output = input_fn()
        if check_fn(output):
            result.passes += 1
        else:
            result.failures.append(output)
    return result


def mean_similarity_pass(similarities: List[float], threshold: float = 0.82) -> bool:
    """
    An alternative to majority vote: instead of thresholding each run
    independently, average the similarity scores across all n runs and
    threshold the mean. This rewards a run that's consistently close to
    the reference even if no single run clears the bar by a wide margin,
    and it punishes one run that's wildly off even if the rest are perfect.
    """
    if not similarities:
        return False
    return (sum(similarities) / len(similarities)) >= threshold


if __name__ == "__main__":
    sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
    from model_client import call_model  # noqa: E402
    from semantic_similarity import is_semantically_equivalent  # noqa: E402

    prompt = "What's the status of my refund for order 4821?"
    reference = "Refund processed for order 4821, 3-5 days"

    result = run_n_times(
        input_fn=lambda: call_model(prompt, temperature=0.7),
        check_fn=lambda output: is_semantically_equivalent(output, reference),
        n=5,
    )
    print(f"pass rate: {result.pass_rate:.2f} ({result.passes}/{result.n})")
    print("majority passed:", result.majority_passed())

The spread itself is the signal worth watching over time, not just the final pass or fail. A scenario that holds at 5-of-5 for three weeks and then drops to 3-of-5 overnight just told you something real changed upstream, a prompt edit, a model version bump, a newly ambiguous tool description, that a single run would have completely hidden.

Pattern Four: Threshold Gating at the Suite Level

Everything above produces a pass rate per scenario. Threshold gating is the last step: rolling those per-scenario rates up into a single suite-level gate with a floor, so a CI run has one clear answer instead of a wall of individual pass rates nobody reads.

The floor itself has to come from somewhere real. Run the suite repeatedly against a version of the system you already trust, look at the noise floor that shows up even when nothing is broken, and set your gate a few points below that measured baseline, not at a round number that looks tidy on a dashboard. A gate set at 95% when your trusted baseline naturally sits at 88% will fail on every run, trusted or not, and a team that lives with a permanently red gate stops looking at it within a month.

Here's a compact gate that rolls per-scenario pass rates into a suite-level pass or fail and returns a process exit code CI can act on:

"""
Threshold gating: gate on a suite-level pass rate against a measured
floor, not on whether a single run happened to pass or fail.

The floor itself should come from a measured baseline (run the suite
repeatedly against a version of the system you already trust, and set
the floor a few points below the noise floor you observe), not from a
round number picked because it looks tidy.

Run with: python lib/threshold_gate.py
"""
import sys
from dataclasses import dataclass
from typing import List


@dataclass
class ScenarioOutcome:
    name: str
    pass_rate: float


def compute_suite_pass_rate(outcomes: List[ScenarioOutcome]) -> float:
    if not outcomes:
        return 0.0
    return sum(o.pass_rate for o in outcomes) / len(outcomes)


def gate(outcomes: List[ScenarioOutcome], floor: float) -> int:
    """
    Returns a process exit code: 0 if the suite clears the floor, 1 if
    it doesn't. Wire this into CI as the last step after the sampling
    runs above have produced a pass_rate per scenario.
    """
    suite_rate = compute_suite_pass_rate(outcomes)
    below_floor = [o for o in outcomes if o.pass_rate < floor]

    print(f"suite pass rate: {suite_rate:.2%} (floor: {floor:.2%})")
    if below_floor:
        print("scenarios below floor:")
        for o in below_floor:
            print(f"  - {o.name}: {o.pass_rate:.2%}")

    return 0 if suite_rate >= floor and not below_floor else 1


if __name__ == "__main__":
    # Example baseline: measured by running this suite 20 times against
    # a known-good build and taking the 10th-percentile pass rate as the
    # floor, not a guess like "95% sounds safe."
    MEASURED_FLOOR = 0.80

    example_outcomes = [
        ScenarioOutcome(name="refund_status_response", pass_rate=0.9),
        ScenarioOutcome(name="off_topic_refusal", pass_rate=1.0),
        ScenarioOutcome(name="ambiguous_order_lookup", pass_rate=0.6),
    ]

    exit_code = gate(example_outcomes, floor=MEASURED_FLOOR)
    sys.exit(exit_code)

Wiring this specific gate into a scheduled or per-PR CI run, alongside the model calls that produce the pass rates in the first place, is its own topic with real cost and latency tradeoffs; our guide to running LLM evals in CI/CD covers that pipeline end to end.

The Practitioner Rule: Too Strict or Too Ambiguous

Here's the rule that resolves almost every "flaky AI test" ticket, stated plainly enough to paste into a Slack thread:

Telling those two apart doesn't require guessing. Run the exact same input against the exact same prompt ten to twenty times and look at the spread of outputs directly, not just at the pass or fail column.

If the outputs agree in meaning but differ in wording (the refund amount is right, the tone is right, only the phrasing changes), your assertion is too strict. You're still checking for a specific string, or a similarity threshold set too high for the amount of legitimate paraphrasing your feature actually produces. The fix lives in the assertion: move from exact match to an invariant or a semantic-similarity check, or loosen the threshold to a number you've actually calibrated against labeled examples instead of guessed at.

If the outputs disagree in meaning or structure (one run confirms the refund, another asks a clarifying question, a third states the wrong amount), your prompt is too ambiguous. No assertion change fixes that, because the model is correctly reflecting an underinstructed prompt back at you in different valid ways. The fix lives upstream: add the missing constraint, specify the output format explicitly, or split an overloaded prompt into two narrower ones. Tightening the assertion around an ambiguous prompt just teaches your suite to accept whichever behavior happened to show up in your sample.

Traditional flaky-test advice (retry it, quarantine it, mark it skip) actively misleads here, because that advice assumes the variance is infrastructure noise sitting on top of a stable, correct system. With AI output, the variance often is the system telling you something true: that the assertion doesn't match what "correct" actually looks like, or that the prompt hasn't pinned down what correct means in the first place. Quarantining that test doesn't remove noise. It removes your only warning.

Where to Go From Here

If you take one thing from this article, take the rule: strict assertion or ambiguous prompt, and the ten-to-twenty-run spread that tells you which one you're looking at. Everything else here, invariants, semantic similarity, sampling, threshold gates, is the tooling that rule needs to actually run in a suite instead of living as a mental checklist.

A reasonable order to build this in: start with invariant assertions on your highest-traffic scenarios, since they're the cheapest and catch the most obvious breakage. Add semantic-similarity checks once you have reference answers worth comparing against. Wrap flaky-by-nature scenarios in n-run sampling before you spend more time debugging a single red run that might not mean anything. Roll everything into a threshold-gated suite once you have enough scenarios that a wall of individual pass rates stops being readable.

None of these four patterns tell you whether the feature actually did the right thing inside your product, only whether its words held up. That's a different, narrower question than whether the right screen loaded, the right record got written, or the right side effect actually fired, which is the layer Autonoma runs against on every preview environment your team ships. This piece is a deep dive under our broader guide to testing generative AI applications, and if the AI feature you're testing is specifically a conversational one, how to test a chatbot covers the surrounding checklist this article's patterns plug into.

Frequently Asked Questions