The Chatbot Testing Checklist (Functional, LLM, and Security)

Most chatbot checklists you'll find are lists of nouns: "test intent recognition," "test context retention," "test for hallucinations." True, and useless on their own. A QA lead reading that list still has to invent the actual mechanic: what input to send, what to assert on, what counts as a pass. That invention step is where most pre-launch reviews quietly stall, or worse, ship with the LLM-specific and security groups untested because nobody wrote down what "test for hallucinations" actually means in a runnable sense.

This is the version that skips the invention step, 34 items across five groups. Every item below carries its how-to-test note inline: what to send, what to check, and where a deeper guide exists if the item deserves one. It condenses our full guide to testing a chatbot; use this page as the checklist you paste into a ticket, and that one when you need the reasoning behind a specific group.

How to Use This AI Chatbot QA Checklist

Not every item belongs at the same cadence, and treating all thirty-plus of them as a single pre-launch pass is how the LLM-specific and performance groups quietly stop getting checked after week one. Split the checklist by when it runs.

The functional and conversational groups are largely a UAT gate: run them once, thoroughly, before launch, and again after any flow or prompt change big enough to warrant a full pass. The LLM-specific, security, and performance groups are different in kind, they test properties of a system that can silently regress on a schedule you don't control (a model provider ships a routine update, a system prompt drifts, latency creeps). Those belong in CI, on every deploy, at whatever depth your API budget allows.

Below the group level, every single line follows the same three-part shape: the item, the concrete how-to-test mechanic, and, where a deeper guide exists in this cluster, a link to it. That shape is deliberate, it's what makes an item something you can hand to an engineer without a follow-up conversation.

For a copy-pasteable set of the actual test cases behind each item, from labeled utterances to seeded entity fixtures, pair this checklist with our chatbot test cases library.

1. Functional Chatbot Testing Checklist (7 Items)

Does the thing work at all. Skip an item here and nothing downstream matters, because a bot that can't extract the right entity will fail every conversational and LLM-specific test for the wrong reason.

• Intent recognition: routes the user's stated goal to the right flow or tool. How to test: build a labeled intent-utterance dataset (paraphrases, typos, slang) and assert routed-intent equality; track a per-intent confusion matrix, not aggregate accuracy.
• Entity extraction: pulls the right slot values (order number, date, product name) from free text. How to test: assert exact match on structured fields against a labeled set, including malformed input like a missing digit or a wrong date format.
• Fallback and no-match handling: out-of-scope or garbled input gets a graceful fallback, never a hallucinated answer. How to test: send out-of-scope prompts and assert the response matches a known fallback pattern, not a fabricated one.
• Handoff to a human: an escalation trigger (explicit request, repeated failure, negative sentiment) actually opens a live-agent session. How to test: assert on the downstream handoff event or ticket, not just that the bot said connecting you to a human.
• Session start and reset: a new session starts clean, and reset actually clears prior context. How to test: assert post-reset responses show no memory of a fact stated only before the reset.
• Integrations and API-backed answers: answers sourced from a live system (inventory, CRM, order status) match its current state. How to test: seed a known backend record, ask about it, and assert the response matches field by field. See chatbot testing tools for mocking options.
• Error states when a downstream service is down: the bot degrades honestly instead of guessing. How to test: mock the dependency's timeout or 500 and assert the response acknowledges the outage rather than inventing an answer.

Watch especially for the gap between what the bot says and what the app actually did, since a confirmation message is not proof an order was placed or a handoff fired, and that behavioral gap is the layer Autonoma tests against the running application rather than the response text.

2. Conversational Testing Checklist (7 Items)

Does it hold a conversation. These items are where a bot that passes every functional test in isolation still falls apart the moment a real user talks to it across more than one turn.

• Multi-turn context retention: a fact stated earlier still applies turns later. How to test: state a fact in turn one, ask unrelated questions in turns two and three, then reference it in turn four and assert correct use.
• Pronoun and referent resolution: "it," "that one," "the first option" resolve to the right prior entity. How to test: after the bot lists options, refer to one by ordinal or pronoun and assert the follow-up targets the correct item.
• Topic switching: a user can abandon one topic mid-flow, start another, and optionally return. How to test: interrupt a flow with an unrelated question, assert it's handled, then resume the original and assert its state wasn't discarded.
• Interruption and correction: "no, I meant the other order" overrides the wrong entity instead of stacking confusion. How to test: trigger a correction mid-flow and assert the corrected entity is used next, not the original.
• Memory across sessions: if the product promises persistent memory, facts carry over; if not, a new session should never leak prior-session facts. How to test: assert both directions, not just the case you expect to pass.
• Tone and persona consistency: the bot's voice doesn't drift across a long or hostile conversation. How to test: run a scripted adversarial conversation and assert against a persona rubric, not by eyeballing tone.
• Graceful ending: the bot recognizes a conversation is over and doesn't loop or re-prompt. How to test: send a closing utterance like thanks, that's all and assert no re-opened flow or extra CTA.

3. LLM Chatbot Testing Checklist (8 Items Most Lists Skip)

This is the group every incumbent checklist either skips or waves at in a paragraph. It's also the reason this post exists: a rule-based bot doesn't have these failure modes, an LLM-backed one has them by default, and none of them show up if you only test the functional and conversational surface above.

• Non-determinism: the same input can legitimately vary in phrasing across runs, but the underlying decision or action must not. How to test: run the same input N times (5 to 10) and assert on a property (extracted entity, tool called, a semantic-similarity threshold), not exact string match. The diagnostic heuristic worth memorizing: a flaky assertion here is either too strict or the prompt is too ambiguous, and telling those two apart is the actual skill. See how to test non-deterministic AI outputs for the full N-run harness.

Here is that harness as a reusable piece. Pass it your own extractor and your own N, and the assertion becomes yours rather than a fixed test. Note what it reports on failure: the distribution of observed values alongside the raw replies, so a red build tells you immediately whether the property moved or only the wording did.

def assert_property_stable(
    message,
    extractor,
    runs=DEFAULT_RUNS,
    property_name="property",
    send=None,
):
    """Send `message` `runs` times and assert `extractor` returns one value.

    Parameters
    ----------
    message : str
        The input to repeat. Use the same string every run; varying the input
        is a different test.
    extractor : callable
        Takes the normalized response dict, returns the property under test.
        Return something hashable and comparable: a string, a tuple, a
        frozenset. Returning the whole reply defeats the purpose.
    runs : int
        How many times to send it. Five catches obvious instability, ten
        catches the tail. Below five you are not sampling, you are guessing.
    property_name : str
        Used in the failure message so a red build names the property.
    send : callable, optional
        `send(message, session_id) -> response dict`. Defaults to the real HTTP
        client. Override it to unit-test your own extractor offline.

    Returns
    -------
    The single stable value, so callers can make further assertions on it.
    """
    if runs < 2:
        raise ValueError("runs must be at least 2 to observe variation")
    sender = send or (lambda text, session_id: send_turn(text, session_id=session_id))

    observed = []
    replies = []
    for index in range(runs):
        response = sender(message, new_session_id("nondet-%d" % index))
        observed.append(extractor(response))
        replies.append(response.get("reply", ""))

    distinct = Counter(repr(value) for value in observed)
    if len(distinct) == 1:
        return observed[0]

    distribution = ", ".join(
        "%s x%d" % (value, count) for value, count in distinct.most_common()
    )
    sample_replies = "\n".join(
        "  run %d: %s" % (i + 1, reply[:160]) for i, reply in enumerate(replies)
    )
    raise AssertionError(
        "Unstable %s across %d runs of %r.\n"
        "The property changed, not just the wording, so this is a real defect "
        "and not a strict-assertion problem.\n"
        "Observed: %s\n"
        "Replies:\n%s"
        % (property_name, runs, message, distribution, sample_replies)
    )


def extract_order_id(response):
    """Property: the entity the bot pulled out of the utterance."""
    return response["entities"].get("order_id")


def extract_tool_names(response):
    """Property: which tools fired, order-insensitive."""
    return frozenset(
        call.get("name") for call in response["tool_calls"] if call.get("name")
    )

The full file adds offline self-tests that prove the comparator actually fails when a property changes, which matters more than it sounds: a stability assertion that can never go red is the easiest thing in this checklist to ship by accident.

• Hallucination and unsupported claims: the bot states a fact it has no source for. How to test: ask questions with a known answer alongside out-of-corpus questions, and assert unsupported claims get flagged or refused, not answered with confidence. See how to test for AI hallucinations for the scoring method.
• Grounding and citation to the actual knowledge source: a claim traces to a real, cited chunk, not a plausible-sounding reference. How to test: assert the citation is a chunk ID that literally appears in the retrieved-context set for that turn. See how to test RAG retrieval for chunk-level assertions.

A runnable version of that assertion, including the negative fixture that matters (a citation formatted perfectly, numbered plausibly, and never retrieved): tests/test_grounding.py.

• Guardrail and refusal behavior: unsafe asks get refused, and in-scope asks don't get over-blocked. How to test: assert refusal on a red-team set and completion on a legitimate adjacent-phrasing set in the same run.
• Prompt-injection resistance (model-level smoke check): an instruction embedded inside retrieved content doesn't get executed as a command. How to test: embed an instruction in a fake document and assert the model ignores it; route the full adversarial suite through the security group below.
• Model-version drift after a silent provider update: behavior that passed yesterday can break because the model changed, not your code. How to test: pin a golden-response regression set and re-run it on a schedule independent of your own deploys.
• Token and context-window truncation in long chats: past the model's context budget, older turns get silently dropped, which can break the multi-turn item above. How to test: run a conversation past the budget and assert the bot summarizes prior context explicitly or fails predictably.
• Streaming-response correctness: partial tokens render correctly and the final message matches a non-streamed call. How to test: assert the concatenated stream equals the non-streaming response, and a mid-stream disconnect doesn't corrupt the next turn's state.

4. Chatbot Security and Privacy Checklist (6 Items)

• Prompt injection: the full red-team corpus across direct, indirect, jailbreak, system-prompt-leak, and data-exfiltration classes. This deserves its own suite, not a spot check. How to test: run the full corpus through a tagged adversarial harness and assert zero bypasses across every class, not an aggregate pass rate. See the runnable prompt injection test suite for the payload corpus and CI gate.
• PII leakage in responses and logs: the bot never emits another user's PII, and logs never capture it unmasked. How to test: run a two-session scenario and scan the response and the log line for PII patterns belonging to a different seeded user.

The two-session scenario, scanning both the response and the log surface, with separator-insensitive matching so a reformatted card number still counts as a leak: tests/test_pii_leak.py.

• Data-retention behavior: data actually gets deleted or anonymized per policy, not just documented as such. How to test: seed a conversation, trigger the retention window (or its test-environment equivalent), and assert the record is gone or anonymized in the store, not just hidden from the UI.
• Authenticated-context bleed between users and sessions: session A's data never leaks into session B. How to test: run two concurrent authenticated sessions with distinct seeded facts and assert B never references A's data.
• Jailbreak resistance: the adversarial, multi-attempt version of the LLM-specific smoke check above. How to test: the same harness, tagged jailbreak, asserting across 5-10 runs per payload, not one.
• Over-permissive tool and function calls: the bot only calls tools it's authorized to call for the current user and scenario. How to test: try to elicit a tool call outside the current user's permission scope and assert it never fires. See testing AI agent tool calls for scope assertions.

5. Performance and Reliability Checklist (6 Items)

• First-token latency vs. full-response latency: users perceive first-token time, not total generation time. How to test: budget and assert both separately, not one blended number.

Two measurements, two thresholds, two independent assertions, so a red build names which half regressed: tests/test_latency_budget.py.

• Concurrency behavior: the bot holds up under N simultaneous sessions without state bleed. How to test: run a concurrency load test and assert correctness and latency budgets both hold under load.
• Timeout and retry handling: a slow model call fails predictably instead of hanging or double-executing a side effect. How to test: inject a timeout and assert the retry doesn't duplicate the underlying action.
• Rate-limit behavior: hitting the provider's rate limit degrades gracefully instead of crashing the session. How to test: simulate a 429 and assert the user-facing behavior, not just the backend log.
• Cost-per-conversation drift: a prompt or model change that quietly increases token usage per conversation. How to test: track token count per conversation type in CI as a regression metric with a budget.
• Graceful degradation when the model provider is slow or down: a full outage falls back to a known-safe state instead of an infinite spinner or a fabricated answer. How to test: black-hole the provider endpoint and assert the fallback fires within a defined budget.

What a Chatbot Testing Checklist Can't Catch

Read this section before you file the checklist as done. It's a coverage map, not a test suite, and treating it as a one-time exercise is exactly how the groups that matter most quietly stop being true.

The functional and conversational groups age reasonably well between full passes, since they mostly regress when someone changes a flow, and someone usually notices. Groups three and five don't have that safety net. Non-determinism, model drift, latency, and cost can all get worse without a single line of your own code changing, because the thing underneath your app changed instead. A checklist item ticked once before launch tells you nothing about whether it's still true today. Only mechanizing it, turning the how-to-test note into an actual assertion that runs in CI, keeps that item honest past week one. See running LLM evals in CI/CD for the pipeline shape.

Mechanizing an item is also where the maintenance question starts, because an assertion written once against last quarter's flow is a checklist item in disguise. On the behavioral items we solved that by not doing it by hand: our Diffs Agent reads each pull request's code diff and adds, updates, or deprecates the end-to-end cases the change affects, so the mechanized items keep describing the product rather than the product as it existed when someone wrote them.

The response-vs-action gap in the functional group deserves the same discipline: a text-only assertion can't catch it, only driving the real app can, which is the specific slice our platform covers rather than trying to infer it from a transcript. Autonoma's Planner reads the codebase behind the bot to derive those cases, and the Executor runs them against the running application, so "an order was placed" becomes an assertion about the order record instead of about the sentence announcing it.

That schedule is the whole operational argument, and it looks like this in practice. The fast job calls nothing, costs nothing, and gates merges. The nightly job is the one that talks to the bot, and it runs on a clock rather than per-push because these items regress when a provider ships an update, not when you push code.

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    # 03:17 UTC daily. Off the hour on purpose: the top of the hour is when
    # every other scheduled job on the runner fleet wakes up, and queue time
    # shows up in your latency numbers as if it were your bot's fault.
    - cron: "17 3 * * *"
  workflow_dispatch:
    inputs:
      model_version:
        description: "Model version to pin for this run (leave blank for the deployed default)"
        required: false
        type: string
      non_determinism_runs:
        description: "How many times to repeat each non-determinism input"
        required: false
        default: "10"
        type: string

permissions:
  contents: read

jobs:
  fast:
    name: Fast checks (every push)
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: pip

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run deterministic checks
        # No endpoint configured, so every test marked `live` is skipped and
        # only the offline harness tests run. Fully deterministic, no spend.
        run: pytest -m "not live" -q

  nightly:
    name: Full suite (nightly and on model bump)
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    timeout-minutes: 30
    environment: chatbot-staging
    env:
      CHATBOT_API_URL: ${{ secrets.CHATBOT_API_URL }}
      CHATBOT_STREAM_URL: ${{ secrets.CHATBOT_STREAM_URL }}
      CHATBOT_API_KEY: ${{ secrets.CHATBOT_API_KEY }}
      CHATBOT_LOG_URL: ${{ secrets.CHATBOT_LOG_URL }}
      CHATBOT_MODEL_VERSION: ${{ inputs.model_version }}
      NON_DETERMINISM_RUNS: ${{ inputs.non_determinism_runs || '10' }}
      LATENCY_SAMPLES: "7"
      TTFT_BUDGET_MS: "1200"
      TOTAL_BUDGET_MS: "6000"

One guard in that nightly job is worth stealing outright: it fails immediately if the endpoint secret is missing. Without it, every live test skips and the job reports green while having tested nothing, which is the most expensive kind of passing build you can own.

Wire the mechanized items into a schedule that matches how they actually regress, a fast subset on every push, the full depth nightly and on model bumps, and the checklist stops being a launch-day artifact and starts being the thing that keeps your chatbot honest for as long as it's in production.

One item on this list stays out of reach of that whole apparatus no matter how well you schedule it, and it's worth ending on: everything above reads the bot's response. That's the right default, it's cheap and it catches most failures, but it means the functional group's most consequential question, whether the order, refund, or handoff the bot announced actually happened, is answered by inference. Autonoma is the layer we'd add for that one, above the checklist rather than in place of it. It doesn't measure latency, score a response, or run your injection corpus, and none of the thirty-plus items here become unnecessary because of it. It drives the real application and asserts on the state the conversation was supposed to leave behind, which is the only way that particular item ever becomes a passing test instead of an assumption.

Frequently Asked Questions