Chatbot Automation Testing: Why Assertions Fail

If you've tried to automate chatbot testing with the same tools you'd use on a login form, you've already hit the wall. The assertion that worked in the demo starts failing on responses that are perfectly correct, just phrased differently. Or worse, it keeps passing on responses that are subtly wrong, because it was never checking meaning in the first place.

This isn't a tooling problem you can fix by picking a different framework. Playwright, Cypress, and every DOM-based test runner assume the thing being tested returns the same output for the same input. A language model breaks that assumption by design. Chatbot automation testing that ignores this fact isn't broken occasionally, it's broken from the first test you write.

The way out is a specific migration, not a specific tool: manual QA to scripted assertions to semantic (LLM-judge) assertions to a CI gate that samples the model N times before it trusts the result. Here's what each stage actually looks like, including the YAML that wires the last one into GitHub Actions.

The Trap of "Just Automate It"

Traditional test automation assumes determinism. Click the same button, the same DOM state comes back, the same assertion passes or fails every time. A chatbot breaks that assumption at the root. Ask it What's your return policy? twice in a row and you'll get two responses that mean the same thing while sharing almost no exact tokens. An assertion like response === "You can return items within 30 days" passes on Tuesday and fails on Wednesday, not because the chatbot got worse, but because it got the same answer right in different words.

Teams that don't confront this early end up in one of two places. Either they give up on automation and go back to a human reading transcripts before every release, which stops scaling past a handful of intents, or they write assertions loose enough to always pass (checking for a single keyword, or just that the response isn't empty), which catches nothing. Neither is chatbot automation testing. Both are theater with a green checkmark attached.

The Migration Path: From Manual QA to a CI Gate

Every chatbot test suite that survives more than two releases goes through the same four stages, whether the team plans it that way or backs into it. Skipping a stage usually means repeating it later, once the gap it was covering shows up in production.

Manual QA

This is where every chatbot starts. Someone opens the chat widget, types the flows they care about, and reads the responses looking for anything wrong: a wrong price, a broken link, a policy that contradicts what support says on the phone. It works fine for one release. By release five, the person doing it has forgotten what they checked two releases ago, and the chatbot has learned a hundred new phrasings since then.

Scripted Assertions

The natural next step is to write it down: turn each manual check into a test that sends the same prompt and asserts something about the reply. This is where most teams stop, and where the exact-match trap sits. An assertion like assert "30 days" in response is really an assertion about a substring, not a fact. Rephrase the knowledge base entry and the substring disappears even though the underlying answer is still correct. Scripted assertions catch obvious breakage (an empty response, a stack trace leaking into the chat window, a reply that blows past your UI's character limit) but they cannot tell you whether the chatbot is still right.

Semantic and LLM-Judge Assertions

The fix is to stop asserting on exact text and start asserting on meaning. Two approaches do this: embed the response and a golden answer with a sentence-embedding model and compare cosine similarity against a threshold, or hand the response and a rubric to a second model and ask it to judge whether the reply satisfies the rubric. Both replace "does this string appear" with "does this response mean the right thing," which is the actual question chatbot regression testing needs answered.

Semantic assertions are necessary, but they only tell you the truth about one sample. A model that answers correctly four times out of five will still pass a single-run check on a lucky day and fail it on an unlucky one. That gap is exactly what the CI gate closes.

The CI Gate

A CI gate takes the semantic assertion, wraps it in a sampling and threshold policy, and wires the result into your pipeline so a pull request that regresses the chatbot cannot merge. This is the stage most vendor content skips entirely: the actual sampling logic, the actual exit code, the actual thing that turns a test script into chatbot automation testing your team can trust rather than a script someone runs manually before a release. The next two sections build that gate.

The Non-Determinism-Safe Automation Core

Everything above describes what to assert. This section is about how to run those assertions safely against a system that won't return the same output twice, which comes down to three ideas working together: n-run sampling, property-based assertions, and threshold gating.

N-Run Sampling

Instead of asking the chatbot once and semantically comparing the answer, ask it N times (5 is a reasonable floor for high-traffic intents, 10 for anything touching money, medical, or legal language) and compare each of the N responses independently against the golden answer. Count how many clear the similarity threshold. Gate on the ratio, not on any single run.

This directly targets a failure mode scripted and single-run semantic assertions can't see: intermittent regression. A prompt-tuning change that drops the model from 100% correct to 80% correct on a given intent won't show up in a single-run test, because most of the time it still looks fine. It shows up immediately in a 5-run sample, because one of the five comes back wrong, consistently, every time you run the suite.

How Autonoma Extends the Gate to the Running App

Everything above verifies the chatbot's message. It does not verify what the chatbot did. A support bot that replies "I've cancelled your subscription" can pass every semantic check and every n-run sample above, whether or not the cancellation actually happened in the billing system sitting behind it. The message-level gate has no way to know, because it never looks past the reply text into the application the chatbot is supposed to be operating on the user's behalf.

That's the layer we built Autonoma for: automating the end-to-end journey, not just the response. Autonoma reads your codebase directly (the routes, the components, the actual flows your chatbot triggers) and plans behavioral end-to-end tests that drive the running chat UI itself, on every pull request. A Planner agent reads the code and plans what a given flow needs to prove. An Execution Agent drives the live UI in an isolated PreviewKit environment, Autonoma's managed preview-environments layer, clicking "confirm cancellation" the way a user would rather than asserting on the reply text alone. A GenerationReviewer separates a genuine product bug from an agent misfire or a plan mismatch. A DiffsAgent keeps the suite current every time a pull request changes the flow, and a HealingAgent adapts the tests when the UI shifts, so a selector change doesn't quietly turn into a false green. Autonoma is open source under our current license (BSL 1.1, converting to Apache 2.0 on 2028-03-23), and it sits above message-level testing, not in place of it. The n-run gate above should still catch a bad reply. This is the layer that catches the bad state behind a reply that looked fine.

Non-Determinism Safeguards: Properties and Thresholds

Sampling tells you whether enough runs agree. Two more ideas decide what "agree" means and what to do about the runs that don't.

Property-Based Assertions

Exact-match and single-value semantic checks both assume there's one correct answer. Real chatbot responses have many correct shapes: a return policy answer might mention 30 days, might mention the returns portal, might mention both, and all are correct. Property-based assertions check for properties instead of values: the response must mention a timeframe, must not promise a refund method the business doesn't support, must not exceed the UI's character budget, must include the escalation phrase for regulated topics. Each property is checked independently and all must hold, but no single property dictates the exact wording the model has to use.

Threshold Gating

Threshold gating is the policy layer that turns "4 of 5 runs passed at 0.85 similarity" into a merge or no-merge decision. Two numbers matter: the similarity threshold per run (how close counts as close enough) and the pass ratio across the N runs (how many of the N need to clear it). Tune both per intent, not globally. A shipping-address confirmation flow might require 5 of 5 runs at 0.9 similarity, because getting it wrong once means a package goes to the wrong address. A conversational small-talk flow might tolerate 3 of 5 at 0.7, because the cost of an occasionally off-tone reply is close to zero.

Here's the harness end to end: it samples the model N times against one golden reference, embeds every response, scores cosine similarity, and exits non-zero the moment the pass ratio misses the configured threshold, the exact shape a CI job needs in order to gate on it.

#!/usr/bin/env python3
"""
n_run_gate.py

An n-run semantic-similarity sampling gate for chatbot regression testing.

For each entry in the golden set, this script calls the chatbot N times,
embeds every response with a sentence-transformer, scores cosine similarity
against the golden reference answer, and passes the entry only if enough of
the N runs clear a similarity threshold. This survives the fact that a
language model returns different wording on every call: a single low-scoring
run doesn't fail the gate, but a chatbot that can't reliably reproduce the
right *meaning* does.

Usage:
    python tests/n_run_gate.py [path/to/golden_set.json]

Exit code is 0 if every golden set entry passes its gate, 1 otherwise, so it
can be used directly as a CI job.

Configuration (environment variables, all optional):
    CHATBOT_ENDPOINT      HTTP endpoint to POST {"prompt": ...} to and read
                          {"response": ...} back from. If unset, the script
                          runs against a small built-in demo bot so the gate
                          is runnable out of the box with no external service.
    N_RUNS                Number of times to sample the bot per prompt.
                          Default 5.
    SIMILARITY_THRESHOLD  Per-run cosine similarity cutoff. Default 0.85.
    PASS_RATIO            Fraction of the N runs that must clear the
                          threshold. Default 0.8 (4 of 5).
"""

from __future__ import annotations

import json
import os
import random
import sys
import urllib.request
from pathlib import Path

from sentence_transformers import SentenceTransformer, util

DEFAULT_GOLDEN_SET_PATH = Path(__file__).resolve().parent.parent / "golden_set.json"
EMBEDDING_MODEL = "all-MiniLM-L6-v2"

N_RUNS = int(os.environ.get("N_RUNS", "5"))
SIMILARITY_THRESHOLD = float(os.environ.get("SIMILARITY_THRESHOLD", "0.85"))
PASS_RATIO = float(os.environ.get("PASS_RATIO", "0.8"))  # 4 of 5 runs

# Demo bot: hand-written paraphrases of each golden_set.json reference answer,
# used only when CHATBOT_ENDPOINT is unset. To test your own chatbot instead
# of this demo, set CHATBOT_ENDPOINT to an HTTP endpoint that accepts
# {"prompt": "..."} and returns {"response": "..."} -- get_bot_response below
# picks that up automatically.
_DEMO_RESPONSES = {
    "What's your return policy?": (
        "You've got 30 days from delivery to send anything back for a full "
        "refund, as long as it's unused and still in its original "
        "packaging. Just start the return from your Orders page and we'll "
        "send a prepaid label."
    ),
    "My order hasn't arrived yet, what do I do?": (
        "Sorry to hear that! Check the tracking link from your shipping "
        "confirmation email first. If there's been no movement for 3 "
        "business days, reach out with your order number and we'll trace "
        "it with the carrier or get a replacement sent out."
    ),
    "Can I change the shipping address after I've placed an order?": (
        "That's possible only before the order ships. Head to Orders, open "
        "the order, and pick Edit Shipping Address. Once tracking shows up, "
        "the address is locked in and we're not able to redirect it."
    ),
    "Do you offer international shipping?": (
        "We do, to more than 40 countries! International delivery runs "
        "7-14 business days, and any customs fees or import taxes are "
        "billed to the customer by the carrier at delivery."
    ),
    "How do I cancel my order?": (
        "You can cancel free of charge as long as it hasn't started "
        "fulfillment. Go to Orders and hit Cancel Order. If it's already "
        "in fulfillment, you'll need to wait for it to arrive and return "
        "it the normal way."
    ),
}

_CLOSERS = [
    "",
    " Let me know if that helps!",
    " Happy to clarify further.",
    " Anything else I can help with?",
]


def get_bot_response(prompt: str) -> str:
    """
    Returns the chatbot's response to a single prompt.

    Point this at your own chatbot by setting the CHATBOT_ENDPOINT
    environment variable to an HTTP endpoint that accepts a POST body of
    {"prompt": "..."} and returns {"response": "..."}. With no endpoint
    configured, this falls back to a small built-in demo bot so the gate is
    runnable without any external service.
    """
    endpoint = os.environ.get("CHATBOT_ENDPOINT")
    if endpoint:
        body = json.dumps({"prompt": prompt}).encode("utf-8")
        request = urllib.request.Request(
            endpoint,
            data=body,
            headers={"Content-Type": "application/json"},
            method="POST",
        )
        with urllib.request.urlopen(request, timeout=10) as response:
            payload = json.loads(response.read().decode("utf-8"))
        return payload["response"]

    # Demo mode: same meaning every run, slightly different wording each
    # time -- exactly the kind of non-determinism the n-run gate exists to
    # survive.
    base = _DEMO_RESPONSES.get(prompt)
    if base is None:
        return "I'm not sure how to answer that one."
    return base + random.choice(_CLOSERS)


def load_golden_set(path: Path) -> list[dict]:
    with open(path, "r", encoding="utf-8") as handle:
        return json.load(handle)


def run_gate(entry: dict, model: SentenceTransformer) -> tuple[bool, list[float]]:
    prompt = entry["prompt"]
    reference_answer = entry["reference_answer"]
    reference_embedding = model.encode(reference_answer, convert_to_tensor=True)

    scores: list[float] = []
    for _ in range(N_RUNS):
        response = get_bot_response(prompt)
        response_embedding = model.encode(response, convert_to_tensor=True)
        similarity = util.cos_sim(reference_embedding, response_embedding).item()
        scores.append(similarity)

    required = max(1, round(N_RUNS * PASS_RATIO))
    passes = sum(1 for score in scores if score >= SIMILARITY_THRESHOLD)
    return passes >= required, scores


def main(argv: list[str]) -> int:
    golden_set_path = Path(argv[1]) if len(argv) > 1 else DEFAULT_GOLDEN_SET_PATH
    golden_set = load_golden_set(golden_set_path)

    print(f"Loading embedding model '{EMBEDDING_MODEL}'...")
    model = SentenceTransformer(EMBEDDING_MODEL)

    required_passes = max(1, round(N_RUNS * PASS_RATIO))
    print(
        f"Gate config: N_RUNS={N_RUNS}, SIMILARITY_THRESHOLD={SIMILARITY_THRESHOLD}, "
        f"PASS_RATIO={PASS_RATIO} ({required_passes}/{N_RUNS} runs must pass)\n"
    )

    all_passed = True
    for entry in golden_set:
        passed, scores = run_gate(entry, model)
        status = "PASS" if passed else "FAIL"
        scores_str = ", ".join(f"{score:.3f}" for score in scores)
        print(f"[{status}] {entry['prompt']!r}")
        print(f"        scores: [{scores_str}]")
        if not passed:
            all_passed = False

    print()
    if all_passed:
        print("Gate result: PASS (all golden set entries cleared their threshold)")
        return 0

    print("Gate result: FAIL (at least one golden set entry missed its threshold)")
    return 1


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Chatbot Regression Testing: Golden Sets and Re-Run Triggers

N-run sampling tells you whether one prompt still passes today. Chatbot regression testing is the practice of re-running every prompt in your suite whenever something upstream of the chatbot changes, and knowing what actually counts as a reason to run it.

Three things justify a full regression run: the model changes (a version bump, a different provider, even a routing change between two models on the same request), the prompt or system instructions change, or the knowledge base changes (a new document gets indexed, an FAQ gets rewritten, a policy gets updated). Miss any of the three and you'll ship a chatbot that passed its last CI run and quietly broke in production the moment support edited a return policy doc that nobody thought to link back to the test suite.

The golden set is what makes any of this repeatable: a fixed collection of prompts paired with reference answers, or reference properties for property-based checks, representing the intents your business actually depends on. It should grow every time a real user finds a gap the suite didn't catch, the same instinct behind any regression suite generally, just applied to conversation instead of clicks or API responses. Version the golden set alongside the code that produces the chatbot's behavior. When the knowledge base changes, diff the golden set's reference answers against the new source documents directly, rather than trusting that the last run's green checkmark still means anything.

This is a narrower job than running LLM evals in CI/CD generally. An evals pipeline scores open-ended generation quality across a broad distribution of prompts. A chatbot regression gate is scoped tighter: specific conversational flows, specific golden answers, specific thresholds per intent, wired to block a specific merge. The techniques overlap; the goals don't.

Wiring the Gate Into CI

None of the above matters if it doesn't block a bad merge before it ships. The workflow below runs on every pull request, executes the n-run harness against the full golden set, and fails the check the moment any prompt's pass ratio drops below its configured threshold:

name: Chatbot Regression Gate

# Runs on every pull request so a bad reply blocks the merge instead of
# shipping and getting caught after the fact. Add this workflow's job as a
# required status check in branch protection to enforce that.
on:
  pull_request:
    branches: [main]

jobs:
  n-run-gate:
    name: n-run semantic similarity gate
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: "pip"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run n-run gate against the golden set
        env:
          # Point this at your staging/preview chatbot endpoint. Left unset
          # here, so the gate runs against the repo's built-in demo bot.
          CHATBOT_ENDPOINT: ${{ secrets.CHATBOT_ENDPOINT }}
          N_RUNS: "5"
          SIMILARITY_THRESHOLD: "0.85"
          PASS_RATIO: "0.8"
        run: python tests/n_run_gate.py golden_set.json
        # A non-zero exit here fails this job, which is what lets GitHub's
        # branch protection treat "n-run semantic similarity gate" as a
        # required check that blocks the merge.

Two details in that workflow matter more than they look. First, it triggers on pull_request, not a push to main, so the gate blocks the merge instead of just alerting someone after the fact. Second, the job exits non-zero on any gate failure, which is what lets GitHub's branch protection treat it as a required check instead of an informational one you can merge past. Skip either detail and what you've built is a dashboard, not a gate.

The Automation Checklist

Before calling a chatbot's test suite "automated," it should clear all of the following, not just the ones that were easy to build first:

• Every assertion checks meaning or properties, never exact strings, because rephrasing a correct answer should never fail a test.
• Every intent that matters is sampled N times per run, with the pass ratio (not a single run) deciding pass or fail.
• Thresholds (similarity and pass ratio) are tuned per intent, tighter for anything involving money, health, or legal exposure.
• A versioned golden set exists, grows when real users find gaps, and gets diffed whenever the knowledge base changes.
• The suite re-runs automatically on model changes, prompt changes, and knowledge base changes, not just on chatbot code changes.
• The CI job triggers on pull requests, exits non-zero on failure, and is wired into branch protection as a required check.
• Something above message level verifies what the chatbot's actions actually did to the application, not only what it said it did.

Clear all seven and you have chatbot automation testing that survives a model that answers the same question five different ways. For chatbot flows that trigger real product actions, pair that gate with Autonoma's behavioral E2E check so each pull request also proves the application reached the state the chatbot reported. Clear fewer, and you have a test suite that's automated in name only, waiting for the first non-deterministic run that slips through.

Frequently Asked Questions