Build a Chatbot Testing Framework in 5 Steps

A working chatbot testing framework needs exactly five components, and skipping any one of them is why most in-house test setups quietly stop catching bugs within a few months. Input layer, golden dataset, assertion layer, regression harness, CI hooks. Miss the assertion layer and every test either false-positives on paraphrased answers or false-negatives on real regressions.

This walkthrough builds a minimal but runnable version of all five in Python, then compares that DIY build against adopting DeepEval or Promptfoo, so the decision to build or adopt comes down to team size and how much metric coverage the product actually needs rather than guesswork. It's one specific, buildable slice of generative AI testing as a whole, and the actual engineering problem this article works through.

What a Chatbot Testing Framework Actually Requires

Strip a chatbot testing framework down to its necessary parts and five things remain, and each one exists to compensate for a specific way conversational systems misbehave that traditional software doesn't.

This is worth separating from a chatbot framework in the other sense of the term. Rasa, Botpress, and similar tools are frameworks for building a chatbot: intents, dialogue state, conversation flows. A chatbot testing framework is a different layer entirely, one that sits outside the bot and checks whether whatever framework built it is actually producing correct answers.

The conversation or input layer is the simplest of the five: a way to send a message, or a multi-turn exchange, into the chatbot and capture what comes back, whether that means calling an API endpoint directly or driving the chat interface itself. Sitting behind it is the golden or eval dataset, a curated set of inputs paired with an expected outcome: a correct intent, a required fact, a tone the response should never violate. Golden datasets are the framework's memory. Without one, every test run is a one-off opinion instead of a comparison against a known-good answer.

The assertion layer is where most homegrown frameworks fail, and it gets its own section below because it's the part exact-match testing can't touch. The regression harness is the piece that turns a one-time evaluation into an actual testing framework: it reruns the golden dataset against every relevant change, on a schedule or on every pull request, and tracks whether pass rates move. CI hooks close the loop by wiring that harness into the same pipeline that already gates merges, so a regression in chatbot behavior blocks a release the same way a failing unit test would.

The golden dataset deserves more care than teams usually give it, because a weak dataset makes every layer built on top of it weaker too. A good one isn't just a handful of happy-path questions pulled from a demo script. It includes edge cases the support team has actually seen: a question phrased two different ways, a request the bot should politely decline, a follow-up in a multi-turn conversation that depends on context from three messages earlier. Treat the dataset like a versioned artifact, not a fixture that gets edited in place. When a new failure mode shows up in production, the fix isn't just patching the bot, it's adding that case to the dataset so the same regression can never sneak back in silently. For a chatbot embedded in a product, this response-level framework pairs with behavioral E2E coverage: it can establish that an answer is sound while Autonoma verifies the user flow the answer is supposed to trigger.

Why Exact-Match Assertions Fail Against Non-Deterministic Output

An assertion like assert response == expected works on a function that returns the same string every time. It fails immediately against an LLM, because the same input can legitimately produce a dozen differently worded correct answers, and none of them will match a hardcoded string byte for byte.

The instinct is to loosen the assertion: check for a substring, check for a keyword, check that the response contains "yes" somewhere. That buys almost nothing. A response can contain every keyword in the expected answer and still be wrong, and a response can omit all of them and still be correct, phrased in different words entirely. Keyword matching is exact-match testing wearing a disguise, and it fails for the same underlying reason, the same gap assertion coverage vs line coverage describes: a check that runs isn't the same thing as a check that verifies anything.

Two mechanisms actually hold up against non-determinism, and they solve different halves of the problem. Semantic similarity scoring embeds both the actual response and the expected response into vector space and measures the distance between them, so a paraphrase that preserves meaning scores high even though the words differ completely. It's fast, cheap, and catches responses that drift far from the expected meaning. What it can't catch is a response that's semantically close but factually or tonally wrong. That's the second mechanism's job: an LLM-judge assertion sends the question, the expected answer, and the actual response to a separate model with a scoring rubric, and asks it to return a verdict (correct, partially correct, or wrong) along with a reason. It's slower and costs a model call per assertion, but it catches what semantic similarity misses: a response that's the right shape and tone but states the wrong price, or answers a different question than the one asked.

A working assertion layer runs both, in sequence. Semantic similarity as a fast first pass, and the LLM judge as the arbiter for anything that doesn't clear a high similarity bar on its own.

Consider a booking assistant asked "can I move my appointment to Thursday." A correct response might say "Sure, I've moved it to Thursday at 2pm" or "Done, your new slot is Thursday afternoon." Both paraphrase the same fact and both score high on semantic similarity against each other, so that layer alone would pass either one without complaint. Now consider a response that says "Sure, I've moved it to Friday at 2pm." It's fluent, on-topic, and structurally almost identical to the correct answer, close enough that a similarity score alone might still pass it. The LLM judge is what catches that specific failure, because it's told what the expected day actually was and asked whether the response matches it, not just whether it sounds like a scheduling confirmation. That's the case that justifies paying for a model call on every assertion instead of relying on embeddings alone.

A Minimal DIY Framework: pytest Plus an LLM Judge

The assertion layer above is small enough to build directly, without adopting anything. Here's the LLM-judge helper: a function that embeds two strings for a similarity score, and a second function that sends a question, an expected answer, and the actual response to a judge model and parses a structured verdict back out.

"""
chatbot_eval.judge
==================

Assertion helpers for testing chatbot responses that are correct in meaning
but never byte-identical to a fixed expected string.

Two functions are exposed:

- semantic_similarity(a, b) -> float
    Embeds both strings with OpenAI's embeddings API and returns the cosine
    similarity between the two vectors, in the range [0.0, 1.0]. A high score
    means the two strings are semantically close even if worded differently.

- llm_judge(question, expected, actual) -> Verdict
    Sends the question, the expected answer, and the actual chatbot response
    to a judge model with a scoring rubric, and returns a structured Verdict
    (label + reason). Use this when semantic similarity alone isn't a strong
    enough signal, e.g. when a wrong response is fluent and on-topic but
    states the wrong fact.

Both functions require the OPENAI_API_KEY environment variable to be set
before they are called. Importing this module does not require the key -
only calling semantic_similarity() or llm_judge() does.
"""

from __future__ import annotations

import json
import math
import os
from dataclasses import dataclass
from typing import Literal, Optional

from openai import OpenAI

EMBEDDING_MODEL = "text-embedding-3-small"
JUDGE_MODEL = "gpt-4o-mini"

VerdictLabel = Literal["correct", "partially_correct", "wrong"]

_client: Optional[OpenAI] = None


def _get_client() -> OpenAI:
    """Lazily construct the OpenAI client so importing this module never
    requires OPENAI_API_KEY to already be set - only calling into it does."""
    global _client
    if _client is None:
        api_key = os.environ.get("OPENAI_API_KEY")
        if not api_key:
            raise RuntimeError(
                "OPENAI_API_KEY is not set. Export it before running any test "
                "that calls semantic_similarity() or llm_judge()."
            )
        _client = OpenAI(api_key=api_key)
    return _client


@dataclass(frozen=True)
class Verdict:
    """Structured result of an LLM-judge call."""

    label: VerdictLabel
    reason: str

    @property
    def passed(self) -> bool:
        """True for a fully correct verdict. `partially_correct` is treated
        as a failure by default - loosen this in the caller if a suite wants
        partial credit to count as a pass."""
        return self.label == "correct"


def _cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float:
    dot = sum(a * b for a, b in zip(vec_a, vec_b))
    norm_a = math.sqrt(sum(a * a for a in vec_a))
    norm_b = math.sqrt(sum(b * b for b in vec_b))
    if norm_a == 0.0 or norm_b == 0.0:
        return 0.0
    return dot / (norm_a * norm_b)


def semantic_similarity(a: str, b: str) -> float:
    """Return a 0.0-1.0 cosine similarity score between two strings' embeddings.

    Raises RuntimeError if OPENAI_API_KEY is unset. Raises openai.OpenAIError
    subclasses on API failures (rate limit, network, auth).
    """
    if not a.strip() or not b.strip():
        return 0.0

    client = _get_client()
    response = client.embeddings.create(model=EMBEDDING_MODEL, input=[a, b])
    vec_a = response.data[0].embedding
    vec_b = response.data[1].embedding

    similarity = _cosine_similarity(vec_a, vec_b)
    # Guard against floating point drift pushing a near-perfect match
    # slightly outside the [0, 1] range.
    return max(0.0, min(1.0, similarity))


_JUDGE_SYSTEM_PROMPT = """You are a strict evaluator for a chatbot testing framework.
You will be given a user question, the expected (golden) answer, and the chatbot's
actual response. Decide whether the actual response is an acceptable answer to the
question given the expected answer as the source of truth.

Score using exactly one of these labels:
- "correct": the actual response conveys the same facts and intent as the expected
  answer, even if phrased completely differently.
- "partially_correct": the actual response is on-topic and not factually wrong, but
  omits something the expected answer requires, or hedges where the expected answer
  is definitive.
- "wrong": the actual response contradicts the expected answer, states a different
  fact (wrong date, wrong price, wrong action), answers a different question, or is
  off-topic.

Respond with ONLY a JSON object of the shape:
{"label": "correct" | "partially_correct" | "wrong", "reason": "<one sentence>"}

Do not include any other text, markdown, or code fences in your response."""


def llm_judge(question: str, expected: str, actual: str) -> Verdict:
    """Ask a judge model to score `actual` against `expected` for `question`.

    Returns a Verdict with a label of "correct", "partially_correct", or
    "wrong", plus a one-sentence reason. Raises RuntimeError if OPENAI_API_KEY
    is unset, or ValueError if the judge model's response can't be parsed as
    the expected JSON shape.
    """
    client = _get_client()

    user_prompt = (
        f"Question: {question}\n\n"
        f"Expected answer: {expected}\n\n"
        f"Actual response: {actual}"
    )

    response = client.chat.completions.create(
        model=JUDGE_MODEL,
        temperature=0,
        messages=[
            {"role": "system", "content": _JUDGE_SYSTEM_PROMPT},
            {"role": "user", "content": user_prompt},
        ],
        response_format={"type": "json_object"},
    )

    raw = response.choices[0].message.content or ""
    try:
        parsed = json.loads(raw)
        label = parsed["label"]
        reason = parsed["reason"]
    except (json.JSONDecodeError, KeyError, TypeError) as exc:
        raise ValueError(
            "llm_judge: could not parse judge model output as the expected "
            f"JSON verdict. Raw output: {raw!r}"
        ) from exc

    if label not in ("correct", "partially_correct", "wrong"):
        raise ValueError(f"llm_judge: unexpected label {label!r} in judge output")

    return Verdict(label=label, reason=reason)

Wiring that helper into pytest takes almost nothing extra, because pytest doesn't care what's inside an assertion, only whether it raises. Here's a test file that loads a small golden dataset inline, calls the chatbot under test, and asserts each response clears both the similarity threshold and the judge's verdict:

"""
tests/test_chatbot_responses.py
================================

Golden-dataset regression tests for a chatbot, using two complementary
assertion layers from chatbot_eval.judge:

1. semantic_similarity(actual, expected) - a fast embedding-based check that
   catches responses that drift far from the expected meaning.
2. llm_judge(question, expected, actual) - a slower model-graded check that
   catches responses that are semantically close but factually or tonally
   wrong (e.g. right shape of answer, wrong day of the week).

Replace `get_chatbot_response()` with a call into your real chatbot (an API
client, a driver for your chat UI, etc). Everything else - the dataset, the
parametrization, and the two-layer assertion - works unchanged.

Run with:
    OPENAI_API_KEY=sk-... pytest tests/test_chatbot_responses.py -v
"""

from __future__ import annotations

import pytest

from chatbot_eval.judge import llm_judge, semantic_similarity

# Similarity below this threshold fails a test outright, before the LLM judge
# is even consulted. Tune this against real production traffic - start
# conservative, log actual scores for the first few weeks, and adjust based
# on where real failures cluster rather than guessing at a round number.
SIMILARITY_THRESHOLD = 0.75

# The golden dataset: each entry is one (question, expected_answer) pair the
# chatbot must handle correctly. Treat this list as a versioned artifact -
# add a new entry every time a real production failure mode is discovered,
# so the same regression can never sneak back in silently.
GOLDEN_DATASET = [
    {
        "id": "reschedule-appointment",
        "question": "can I move my appointment to Thursday",
        "expected": "Sure, I've moved your appointment to Thursday.",
    },
    {
        "id": "cancel-appointment",
        "question": "I need to cancel my appointment",
        "expected": "Your appointment has been cancelled. Let me know if you'd like to rebook.",
    },
    {
        "id": "business-hours",
        "question": "what time do you open",
        "expected": "We're open from 9am to 6pm, Monday through Friday.",
    },
    {
        "id": "refund-policy",
        "question": "can I get a refund if I cancel within 24 hours",
        "expected": "Yes, cancellations made at least 24 hours in advance are fully refundable.",
    },
    {
        "id": "out-of-scope-request",
        "question": "can you write my resignation letter for me",
        "expected": "I can only help with scheduling and account questions for this business, so I can't help with that.",
    },
]


def get_chatbot_response(question: str) -> str:
    """Stub chatbot client. Replace this with a call to your real chatbot,
    e.g. `requests.post(CHATBOT_API_URL, json={"message": question}).json()["reply"]`
    or a driver call against your chat UI.

    This stub echoes a canned response per question so the test file is
    runnable end-to-end (against the real OpenAI-backed assertion layer)
    before any real chatbot integration exists.
    """
    canned_responses = {
        "can I move my appointment to Thursday": "Done, your new slot is Thursday afternoon.",
        "I need to cancel my appointment": "Your appointment has been cancelled. Want me to help you rebook?",
        "what time do you open": "We open at 9am and close at 6pm on weekdays.",
        "can I get a refund if I cancel within 24 hours": (
            "Yes, as long as you cancel 24 hours ahead of time, you'll get a full refund."
        ),
        "can you write my resignation letter for me": (
            "That's outside what I can help with here, I'm only set up for "
            "scheduling and account questions."
        ),
    }
    return canned_responses.get(question, "I'm not sure how to help with that.")


@pytest.mark.parametrize(
    "case",
    GOLDEN_DATASET,
    ids=[case["id"] for case in GOLDEN_DATASET],
)
def test_chatbot_response_matches_golden_answer(case: dict) -> None:
    question = case["question"]
    expected = case["expected"]

    actual = get_chatbot_response(question)

    similarity = semantic_similarity(actual, expected)
    assert similarity >= SIMILARITY_THRESHOLD, (
        f"[{case['id']}] semantic similarity {similarity:.2f} is below "
        f"threshold {SIMILARITY_THRESHOLD}.\n"
        f"  question: {question}\n  expected: {expected}\n  actual:   {actual}"
    )

    verdict = llm_judge(question=question, expected=expected, actual=actual)
    assert verdict.passed, (
        f"[{case['id']}] LLM judge returned '{verdict.label}': {verdict.reason}\n"
        f"  question: {question}\n  expected: {expected}\n  actual:   {actual}"
    )

Run it with pytest tests/test_chatbot_responses.py -v and every entry in the golden dataset becomes its own test case, which means a new failure shows up as one named test going red, not a general sense that something regressed. This is the entire DIY framework: a dataset, two assertion functions, and pytest doing the parametrization and reporting it already does well. Nothing here demands a dedicated evaluation platform. It demands roughly a hundred lines of code and a place to keep the golden dataset. It also stays intentionally focused on response quality, which is why it complements Autonoma's end-to-end coverage rather than duplicating it.

The threshold on the similarity check is the one number in this whole setup worth tuning deliberately. Set it too low and paraphrases that changed meaning slip through as passes. Set it too high and correct answers phrased unusually start failing a check that was supposed to tolerate exactly that kind of variation, which is how a team ends up distrusting its own test suite and starts ignoring red results. Start conservative, log the actual similarity scores alongside every pass and fail for the first few weeks, and adjust the threshold based on where real failures cluster rather than guessing at a round number upfront.

How Autonoma Covers What This Framework Doesn't Test

Everything above operates at the response level. It asks whether a given input produces an acceptably correct, on-brand piece of text. It says nothing about what happens next inside the actual product: whether tapping the button the chatbot just suggested fires the right API call, whether the conversation state survives a page reload, whether the chatbot's structured response (a list of options, a booking confirmation, a rendered card) actually appears correctly in the UI a real user is looking at. A response can pass every assertion in the golden dataset above and still lead to a broken button, because nothing in that framework ever clicked it.

That's a different layer of testing, and it's the layer Autonoma covers. It reads the codebase to plan end-to-end test cases and drives the actual chat interface in a live Autonoma PreviewKit preview environment, verifying real application behavior around the conversation rather than just the text the model returned. It doesn't replace the assertion layer above, an open-source runtime driving UI behavior isn't scoring semantic similarity or running an LLM judge against a response string, it picks up exactly where that layer stops: at the moment a correct response is supposed to trigger something real in the product. Run the response-level framework to catch a wrong answer. Run the behavioral layer to catch a right answer that the UI still fails to act on.

Build vs Adopt: DIY, DeepEval, and Promptfoo

Building the minimal version above costs an afternoon. Adopting DeepEval or Promptfoo costs an integration and a subscription to their conventions. Neither is universally right, and the decision comes down to four things: how much it costs to set up, how many metrics come built in versus need to be written by hand, how much ongoing maintenance the framework itself demands, and how cleanly it drops into a CI pipeline.

Both DeepEval and Promptfoo are open source, so open source chatbot testing doesn't require choosing between "build it yourself" and "pay for a platform." Both can be self-hosted and inspected the same way the DIY approach can, the difference is how much of the assertion logic ships already written.

The DIY approach wins on cost and control: nothing to learn beyond pytest, and every metric means exactly what the team decided it means, because the team wrote it. DeepEval and Promptfoo both win on breadth: dozens of metrics that would take weeks to write and validate from scratch (answer relevancy, faithfulness, hallucination detection, and more) ship already implemented and maintained by people whose only job is keeping them accurate as models change. That maintenance is the real trade. A DIY judge function needs its prompt and its threshold revisited by hand every time a model upgrade shifts scoring behavior. An adopted framework's maintainers do that work across their whole user base at once, which is either a meaningful advantage or an irrelevant one depending on how many metrics the product actually needs beyond correctness and tone.

Deciding Between Building and Adopting

Team size is usually the strongest signal. A single engineer covering chatbot QA on the side benefits most from DIY, because there's no team to coordinate around a shared tool's opinions, and the entire framework fits in one person's head. A larger team running dozens of conversational flows across a product benefits from the opposite: an adopted framework means every contributor writes tests against the same vocabulary of metrics instead of reinventing assertion helpers per feature.

Metric needs settle most of the remaining cases. A team that only needs to check for correctness and tone rarely needs more than the DIY judge described above. A team that also needs to measure hallucination rate, retrieval faithfulness for a RAG-backed bot, or toxicity across a large volume of adversarial inputs is reimplementing a meaningful slice of DeepEval or Promptfoo from scratch if it stays DIY, at which point adopting is usually the cheaper option, not the more expensive one.

A useful test for where a specific team lands: count how many distinct things the golden dataset needs to check beyond "was this the right answer." One or two, correctness and maybe tone, and the DIY judge covers it without much extra code. Five or six, correctness, faithfulness to a knowledge base, toxicity, refusal behavior on restricted topics, tone across multiple brand voices, and the team is now maintaining several bespoke judge prompts that an adopted framework already ships tested and versioned. The number of metrics, not the number of engineers, is usually what tips the decision once a team gets past the "one person, one bot" stage.

Whichever side of that decision the team lands on, response-level testing is answering one question: is this answer correct. It was never going to answer the other one: does the product actually behave correctly once that answer reaches a user. That's the gap Autonoma closes, driving the real chat interface end to end so a correct response that the UI fails to act on gets caught before it ships, not after. Build the assertion layer, or adopt one. Either way, pair it with a behavioral layer that watches what happens after the chatbot replies.

FAQ

Frequently Asked Questions