Chatbot Testing Tools: Open-Source and Commercial (2026)

Search "chatbot testing tools" right now. The top-ranking pages assume your chatbot is a decision tree: a script that walks a fixed menu of intents, checks the bot said the expected line, and calls it tested. That approach was written for a world that mostly doesn't exist anymore. Most production chatbots today are a thin UI wrapped around an LLM, which means the thing you actually need to verify doesn't behave like a decision tree. It behaves like a probabilistic function that can phrase the same correct answer four different ways, and phrase a confidently wrong one just as smoothly.

Every page currently ranking for this exact phrase picks a side and doesn't say so. The scripted-testing pages check the flow and never touch the model's actual output quality. The eval-framework pages score the model and rarely mention what happens in your product after the model responds. Nobody publishes the honest split, or tells a practitioner which side they actually need first.

Two Categories of Chatbot Testing Tools, and Why Vendors Blur Them

The split exists because chatbots evolved through two distinct eras, and the tooling never fully caught up with the second one. The first era was rule-based and intent-driven: a user's message got classified into one of a fixed set of intents, and the bot picked a pre-written response from a matching branch. Testing that kind of bot is deterministic UI testing with extra steps. You script the input, assert the exact output string, and the assertion either passes or fails the same way every single run.

The second era replaced the fixed branches with a language model. The input is still a user message, but the output is now generated, not selected. Two identical inputs can produce two differently worded but equally correct responses, and a single character of context drift can produce a differently worded and subtly wrong one. An assertion built for the first era, exact string match, breaks immediately against the second, not because the bot is broken but because determinism was never the property to test for in the first place.

Most tools sold as "chatbot testing tools" were built for the first era and retrofitted, at best, for the second. That is not a knock on any single vendor. It's a description of category drift: the market kept the same search term while the underlying product changed underneath it. The practical result for anyone evaluating tools today is that you have to know which era a given tool was designed for before its feature list means anything.

This matters because the failure modes are completely different, and a tool built for one won't surface a failure from the other. A first-era bot fails loudly: the wrong branch fires, the exact string doesn't match, the script goes red in an obvious, reproducible way. A second-era bot fails quietly. The response reads as fluent, confident, and plausible, and it's still wrong, or it's right in wording and wrong in the state it implies. An assertion library built to catch the first failure mode is structurally blind to the second, because it was never asked to reason about correctness, only about equality.

That's also why a tool's marketing copy is a poor signal here. A page that says "automated chatbot testing" without saying which era it assumes is telling you nothing about whether it can catch the failure your bot actually produces. The honest question to ask any vendor, including the ones in this article, is narrower: what specific failure mode does this tool catch, and what does it structurally miss.

Legacy Scripted-Bot Testing vs. LLM-Native Eval

Neither branch is obsolete. A bank's IVR menu and a support widget's canned-response fallback still exist, and they still need scripted regression coverage. The question is not which branch is better. It's which branch matches what your chatbot actually does, and for teams running an LLM in production, that's almost always the right-hand branch, sometimes both.

Read those two tables side by side and a pattern falls out immediately: every single row, in both tables, evaluates the response. Botium checks whether the response string matched the script. Promptfoo checks whether the response passed an assertion. Ragas checks whether the response was grounded in retrieved context. None of them, not one row across nine tools, checks whether the response actually did anything once it left the chat window. If the bot says "your refund has been processed," none of these tools open the account page and confirm a refund record exists. If the bot says "I've escalated you to a human," none of them confirm a ticket got created. The response was correct on paper and nothing behind it happened.

The Two Strongest Open-Source Options, Running

Of the six LLM-native tools above, Promptfoo and DeepEval cover the widest practical ground for a team getting started, and both are free, self-hostable, and git-friendly. Promptfoo's whole model is a declarative test matrix: define your prompts, your test cases, and your assertions in one config file, and it runs the matrix and reports pass/fail per cell. Here's a config that checks a support bot's refund-policy response for both a keyword and a semantic similarity assertion, so a correct answer phrased two different ways still passes:

# Promptfoo config: refund-policy response quality for a support chatbot.
#
# Prerequisites:
#   npm install -g promptfoo
#   export OPENAI_API_KEY=sk-...   # used both to generate responses and to
#                                  # score the `similar` (semantic) assertion
#
# Run:
#   promptfoo eval -c promptfooconfig.yaml
#
# Two test cases ask the same underlying question in different words. Both
# should pass: the keyword assertion checks the "30 days" figure is present,
# and the semantic-similarity assertion checks the meaning matches a known-
# good reference answer even when the wording differs.

description: "Refund-policy response quality for a support chatbot"

prompts:
  - |
    You are a customer support assistant for an online retailer. Answer the
    customer's question about the refund policy accurately and concisely,
    using only the policy below.

    Refund policy: Items may be returned within 30 days of purchase for a
    full refund to the original payment method, provided they are unused and
    in their original packaging. Refunds are processed within 5-7 business
    days after the returned item is received and inspected. Final sale items
    are not eligible for return.

    Customer question: {{question}}

providers:
  - openai:gpt-4o-mini

defaultTest:
  assert:
    - type: icontains
      value: "30 days"
    - type: similar
      value: "You can return items within 30 days of purchase for a full refund, as long as they're unused and in original packaging."
      threshold: 0.75

tests:
  - description: "Plainly phrased refund question"
    vars:
      question: "How long do I have to return an item for a refund?"

  - description: "Differently phrased but equally correct question"
    vars:
      question: "I bought something last week and it doesn't fit. What's the cutoff for sending it back?"

DeepEval takes a different shape: pytest-style test functions with LLM-specific assertion helpers, which means it slots directly into a suite your team is probably already running in CI. Here's an assertion that scores a chatbot's response for both answer relevancy and factual consistency against a source document, and fails the build if either metric drops below threshold:

"""
DeepEval assertions for a support chatbot's refund-policy response.

Prerequisites:
    pip install deepeval pytest
    export OPENAI_API_KEY=sk-...   # DeepEval's default metrics use an LLM judge

Run:
    deepeval test run test_chatbot_response.py

Scores a chatbot's response with two metrics against a source policy
document: AnswerRelevancyMetric (does the response actually address the
question) and FaithfulnessMetric (does the response stay factually
consistent with the provided context, instead of inventing details). The
test run fails if either metric drops below its threshold.
"""

import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

SOURCE_POLICY_DOCUMENT = (
    "Refund Policy: Items may be returned within 30 days of purchase for a "
    "full refund to the original payment method, provided they are unused "
    "and in their original packaging. Refunds are processed within 5-7 "
    "business days after the returned item is received and inspected. "
    "Final sale items are not eligible for return or refund."
)


def get_chatbot_response(question: str) -> str:
    """
    Stand-in for the real chatbot call. Replace this with an actual
    invocation of your support bot (API call, model call, etc.) - the rest
    of the test is unaffected either way.
    """
    responses = {
        "What is your refund policy?": (
            "You can return any unused item within 30 days of purchase for "
            "a full refund to your original payment method, as long as it's "
            "in its original packaging. Refunds are processed within 5-7 "
            "business days after we receive and inspect the returned item. "
            "Final sale items can't be returned."
        ),
    }
    return responses[question]


@pytest.mark.parametrize("question", ["What is your refund policy?"])
def test_chatbot_refund_response_quality(question: str):
    actual_output = get_chatbot_response(question)

    test_case = LLMTestCase(
        input=question,
        actual_output=actual_output,
        retrieval_context=[SOURCE_POLICY_DOCUMENT],
    )

    answer_relevancy = AnswerRelevancyMetric(threshold=0.7)
    faithfulness = FaithfulnessMetric(threshold=0.7)

    assert_test(test_case, [answer_relevancy, faithfulness])

Both examples above prove the response was well-formed. Neither one, by design, opens a browser and checks what the application did after the bot said it. That's not a gap in Promptfoo or DeepEval specifically. It's a gap in the category. Response-level eval and behavioral verification are answering two different questions, and a chatbot testing stack that only answers the first one is missing the failure mode that actually reaches production: the bot says the refund is processed, and it isn't.

Both tools are also cheap to adopt for exactly the reason they're limited: they don't need a running application, a browser, or a deployed environment. A config file and a CI job are enough. That's a genuine advantage for layer-one feedback speed, catching a regressed prompt in the seconds it takes a pull request to build, well before anyone would want to pay the cost of spinning up a full Autonoma PreviewKit preview environment to check it. The tradeoff is exactly the one this article keeps returning to: fast and cheap at the response layer, silent at every layer above it.

Where Each Tool Sits Across Four Testing Layers

Laid out as layers instead of a vendor list, the gap gets easier to see. A mature chatbot testing stack has four layers, and every tool discussed above lives in exactly one of the first two.

Layer one is the fastest feedback loop: prompt-level assertions that run in seconds against a config file, catching regressions before a PR merges. Layer two widens the lens to full eval sets, scoring hundreds of scripted conversations for drift, bias, and grounding, usually on a schedule rather than per-commit. Layer four closes the loop in production, tracing real conversations and flagging drift after the fact, which matters but arrives too late to stop a bad release. Layer three is the one this entire tool list leaves alone: did the correct-sounding response actually do the thing it claimed to do, inside the real application, before the release shipped.

Teams usually build layers one, two, and four in roughly that order, because each one is a natural next step from the last: assertions come first because they're cheapest, eval sets come next because they scale the same idea, and production monitoring comes last because it needs real traffic to monitor. Layer three doesn't fit that progression. It doesn't get cheaper or more obviously necessary as the other three mature, so it's the layer teams skip by default rather than the layer they consciously decide against, which is precisely why it's still empty on most chatbot testing tools roundups a full generation into the LLM-chatbot era.

How Autonoma Tests What Happens After the Bot Responds

The pattern this article has been building toward is specific: response-level correctness and product-level behavior are independent claims, and the entire chatbot testing tools market, legacy and LLM-native alike, has concentrated its tooling on the first claim. A bot can pass every assertion in a Promptfoo matrix, clear every Giskard safety scan, and still ship a refund flow that never touches the database, because none of those tools ever open the application the chatbot is embedded in.

That's the layer we built Autonoma to cover, and it's the same end-to-end behavioral testing discipline Autonoma already runs against conventional web applications, pointed at AI-native product surfaces instead. Autonoma doesn't score models, compute eval metrics, or replace Promptfoo or DeepEval in your CI pipeline. Its Planner agent reads the actual application your chatbot is embedded in, plans test cases against the real user flows a conversation can trigger, and generates the database state each scenario needs. The Execution Agent then drives the live product in an Autonoma PreviewKit preview environment, sending the chatbot a message and following through on whatever the response claims happened next: does the order status page reflect the change, does the escalation actually create a ticket, does the confirmed action persist. GenerationReviewer classifies each run's result as a genuine product bug, an agent error, or a mismatch between the test and the current flow, so a failure means something specific instead of just red. HealingAgent adapts the test when the surrounding UI changes shape, and DiffsAgent keeps the suite aligned as the codebase evolves, maintaining coverage on every pull request without anyone rewriting scripts by hand.

Mapped against the four layers above: Autonoma doesn't compete for layer one or two, where Promptfoo, DeepEval, Giskard, Ragas, Deepchecks, and LangSmith already do real, specific work worth keeping. It fills layer three, the one every tool in this comparison quietly skips, by verifying the consequence of the response rather than the wording of it. PreviewKit gives that behavioral layer a managed per-PR environment, so the check can run against the same product change the response-level suite is gating.

Chatbot Testing Services vs. Doing It In-House

The other half of this SERP is entirely managed-QA sales pages, firms like Testlio, TestingXperts, and DeviQA offering chatbot testing services as a staffed engagement rather than a tool you run yourself. That's a legitimate option for a specific situation, and a poor fit for most others, so it's worth being honest about the tradeoff instead of pretending one path is universally right.

Chatbot testing services make sense when a team has no in-house QA capacity at all, needs broad manual coverage across many conversation paths and languages fast, or is running a one-time pre-launch audit rather than building continuous coverage. What you're buying is human hours and existing test-writing infrastructure, not a permanent piece of your engineering stack. The tradeoff is control and speed: a services engagement runs on the vendor's cadence, not your release cadence, and the coverage typically doesn't run again automatically the next time you change a prompt or a flow.

Building the stack in-house, Promptfoo or DeepEval for layer one, an eval framework for layer two, costs engineering time up front but runs on every commit, in your CI, on your schedule, for the cost of compute rather than a day rate. For a team already shipping an LLM-backed chatbot with any regularity, that ongoing cadence is usually worth more than the breadth a one-time services engagement provides. The pragmatic middle path many teams land on: build the automated layers in-house for continuous coverage, and bring in a services engagement only for the periodic deep audits, a new language launch, a compliance review, a major flow redesign, that genuinely benefit from dedicated human hours instead of another CI run.

The cost comparison also isn't static over the life of a product. A services engagement scales roughly linearly with conversation volume and language count, since it's ultimately paying for reviewer time, while an in-house eval suite scales with compute, which gets cheaper per run as the team's tooling matures. Early on, before a chatbot has enough real usage to justify the engineering investment, a services audit can be the more rational spend. Past that point, the math tends to flip, and the teams still paying per-audit for coverage they could run automatically every night are usually the ones who never revisited the decision after making it once.

None of this changes what a services engagement or an in-house eval suite actually verifies, though. Both are still operating at the response layer. A thorough manual audit from a QA vendor can catch a wrong or unsafe answer just as well as an automated eval set can. Neither one, on its own, confirms the application did what the answer said it would.

For teams building the automated pieces of this stack themselves, how to test a chatbot walks through the practical setup for layers one and two, and a chatbot testing framework covers how to structure that suite so it scales past a handful of scripted conversations. Once those layers are running reliably, chatbot automation testing covers the additional layer of wiring this suite into a CI gate so it runs on every pull request instead of only when someone remembers to run it manually.

Frequently Asked Questions