How to Test for AI Hallucinations: A Code-First Guide

How to test for AI hallucinations means running two complementary kinds of checks: a reference-based check that compares an answer against a labeled, known-good response for a curated regression set, and a reference-free check that verifies an answer is grounded in whatever context it was actually given, the kind that scales to live production traffic. Both need runnable assertions, not just metric names, to catch a hallucination before it reaches a user.

The support bot answered in under a second. It quoted the refund policy correctly, structured the response cleanly, and closed with a friendly line about being happy to help further. It also stated, with the same even confidence as everything else in the message, that a $10 restocking fee applies. No restocking fee exists in the company's policy, in the context the model was given, or anywhere in reality. Every test the team had passed: the response answered the question asked, the tone matched the brand voice, latency was fine, the JSON parsed. Nothing about the failure looked like a bug, because by the standards of a conventional test suite, it wasn't one. It was a hallucination, and the suite that would have caught it did not exist yet.

What Counts as a Hallucination (and What Doesn't)

"Hallucination" gets used as a catch-all for "the AI said something wrong," which is too loose a definition to test against. Three distinct failure modes hide inside that one word, and they need different checks.

An ungrounded claim is a statement the model was not given any basis for, regardless of whether it happens to be true in the world. The restocking fee above is ungrounded: nothing in the retrieved policy text supports it, whether or not some other store somewhere charges one. A factually false claim is wrong independent of context, the kind of error a reference-based check against a known-good answer catches even without a retrieval step: a wrong date, a wrong formula, a wrong version number. An unsupported extrapolation is subtler than both. The model takes something the context actually says and reasons one step past it, inferring a general policy from a specific example, or assuming a rule applies to a case the context never covered. It often sounds the most reasonable of the three, which is exactly why it is the easiest one to ship by accident.

The distinction matters because it decides which check catches which failure. Groundedness and faithfulness checks are built to catch ungrounded claims and, with more care in the rubric, unsupported extrapolation. They are not built to catch a factually false claim that happens to match the given context, because a groundedness check only ever asks "does the context support this," never "is the context itself correct." Keep that boundary explicit before writing a single assertion, or you'll end up debugging a faithfulness score that was doing exactly what you asked it to do. If you're still mapping out the wider surface a genAI feature needs covered, testing generative AI applications is the broader guide this hallucination work sits inside.

Reference-Based vs Reference-Free Checks

The first fork in the road is whether you have a labeled, known-good answer for the input you're testing. If you do, a reference-based check compares the new answer against that golden reference, usually with semantic similarity rather than exact string match, since two correct answers can be worded completely differently. That comparison is precise and cheap to compute, but it only exists for the inputs someone bothered to label, which means it lives entirely inside a curated regression set. It has nothing to say about the live traffic your users are actually sending, because none of that traffic has a labeled answer waiting for it.

A reference-free check drops the labeled-answer requirement entirely. Instead of asking "does this match the known-good answer," it asks "is this answer supported by whatever context the model was actually given." That reframing is what makes it viable on open-ended, unlabeled production traffic: it needs the retrieved context (which you always have, by construction) and the answer (which you always have too), and nothing else. The tradeoff is real: a reference-free check can tell you an answer is well-grounded in its context and still have no opinion on whether that grounded answer was the right one to give, since it never compares against what a correct answer should have said.

Neither path is strictly better. Reference-based checks need labels and don't scale; reference-free checks scale but can't tell you if the grounded answer was the one you wanted.

Dimension	Reference-Based	Reference-Free
Ground truth needed	Yes, a labeled answer	No, only given context
Scalability	Limited to the labeled set	Scales to any input
Prod-traffic viability	Not viable, no labels exist	Viable, runs on live traffic
What it catches	Drift from the golden answer	Claims unsupported by context
What it misses	Anything outside the golden set	Whether it's the answer you wanted

Here's the reference-based version, semantic similarity against a golden set instead of exact match, gated with a threshold and margin rather than exact equality:

"""
Reference-based check: compare a new answer against a known-good reference
answer using semantic similarity, not exact match. Needs a labeled ground
truth per question, so it fits a curated regression set, not open-ended
live traffic.

semantic_similarity takes an injected embed_fn so this module has no hard
dependency on any specific embedding backend. In production, embed_fn is
typically a thin wrapper around a real embedding model (for example
sentence-transformers, or an embeddings API). In this test file, embed_fn
is a small hand-built lookup, so the suite runs offline with no network
access and no model download. Swap it for a real embedding call in your
own suite; the assertion logic in semantic_similarity does not change.
"""
import pytest

GOLDEN_SET = {
    "What is your refund window?": "You can request a refund within 30 days of purchase.",
    "How long do refunds take to process?": "Refunds are issued within 5 to 7 business days.",
}

SIMILARITY_THRESHOLD = 0.75  # margin, not exact equality

_FAKE_EMBEDDINGS = {
    "You can request a refund within 30 days of purchase.": (1.0, 0.9, 0.1),
    "Refunds can be requested up to 30 days after you buy.": (0.95, 0.85, 0.1),
    "Refunds are issued within 5 to 7 business days.": (0.1, 0.2, 1.0),
    "Expect your refund within 5-7 business days.": (0.15, 0.25, 0.95),
    "Our headquarters are located in Austin, Texas.": (0.0, 0.0, 0.0),
}


def stub_embed(text: str):
    """A tiny hand-built lookup standing in for a real embedding model."""
    return _FAKE_EMBEDDINGS.get(text, (0.0, 0.0, 0.0))


def semantic_similarity(a: str, b: str, embed_fn) -> float:
    """Cosine similarity between two strings, using an injected embedding
    function so the caller controls the actual model."""
    vec_a = embed_fn(a)
    vec_b = embed_fn(b)
    dot = sum(x * y for x, y in zip(vec_a, vec_b))
    norm_a = sum(x * x for x in vec_a) ** 0.5
    norm_b = sum(y * y for y in vec_b) ** 0.5
    if norm_a == 0 or norm_b == 0:
        return 0.0
    return dot / (norm_a * norm_b)


@pytest.fixture
def generate_answer():
    """Swap this for a call into your actual app or agent under test."""
    canned = {
        "What is your refund window?": "Refunds can be requested up to 30 days after you buy.",
        "How long do refunds take to process?": "Expect your refund within 5-7 business days.",
    }

    def _stub(question: str) -> str:
        return canned[question]

    return _stub


@pytest.mark.parametrize("question,reference", GOLDEN_SET.items())
def test_answer_matches_reference_semantically(question, reference, generate_answer):
    candidate = generate_answer(question)
    score = semantic_similarity(candidate, reference, stub_embed)
    assert score >= SIMILARITY_THRESHOLD, (
        f"Answer drifted too far from the golden reference (similarity {score:.2f}): {candidate!r}"
    )


def test_unrelated_answer_fails_similarity():
    candidate = "Our headquarters are located in Austin, Texas."
    reference = GOLDEN_SET["What is your refund window?"]
    score = semantic_similarity(candidate, reference, stub_embed)
    assert score < SIMILARITY_THRESHOLD

And here's the reference-free version, a groundedness check that needs no labeled answer at all, only the context the model was actually given:

"""
Reference-free groundedness check: no labeled "known-good" answer required.
Instead, checks whether the answer is entailed by whatever context the
model was actually given, using an injected NLI-style function. Because it
needs no ground truth, this is the check you can run against live,
open-ended production traffic, not just a curated golden set.

nli_fn is injected the same way embed_fn is injected in
test_reference_based_check.py, so this module has no hard dependency on any
specific NLI backend. In production, nli_fn is typically a cross-encoder
NLI model or a judge-model call. In this test file it is a hand-built
lookup, so the suite runs offline with no network access.
"""
GROUNDEDNESS_THRESHOLD = 0.5  # entailment probability, margin not exact equality

_FAKE_NLI_TABLE = {
    (
        "Our refund policy allows returns within 30 days of purchase with a receipt.",
        "You can return your purchase within 30 days if you kept the receipt.",
    ): 0.93,
    (
        "Our refund policy allows returns within 30 days of purchase with a receipt.",
        "You can return your purchase within 30 days, minus a 15% restocking fee.",
    ): 0.20,
}


def stub_nli(context: str, answer: str) -> float:
    """A tiny hand-built lookup standing in for a real NLI/entailment model."""
    return _FAKE_NLI_TABLE.get((context, answer), 0.0)


def groundedness_score(context: str, answer: str, nli_fn) -> float:
    """Return the probability that `answer` is entailed by `context`."""
    return nli_fn(context, answer)


def test_grounded_answer_passes():
    context = "Our refund policy allows returns within 30 days of purchase with a receipt."
    answer = "You can return your purchase within 30 days if you kept the receipt."
    assert groundedness_score(context, answer, stub_nli) >= GROUNDEDNESS_THRESHOLD


def test_ungrounded_answer_fails():
    context = "Our refund policy allows returns within 30 days of purchase with a receipt."
    # Invents a detail nowhere in the context: a 15% restocking fee.
    answer = "You can return your purchase within 30 days, minus a 15% restocking fee."
    assert groundedness_score(context, answer, stub_nli) < GROUNDEDNESS_THRESHOLD

Most teams end up running both. The reference-based suite guards a curated set of high-stakes, known scenarios (the ones a human already wrote a correct answer for). The reference-free suite runs continuously against everything else, since it's the only check of the two that doesn't need someone to have labeled the input first.

RAG Faithfulness: Keeping the Answer Grounded in Retrieved Context

Faithfulness is the reference-free check applied specifically to retrieval-augmented generation: given the chunks your retriever pulled back, does the generated answer stay inside what those chunks actually support? The metric name shows up everywhere in the current SERP. The runnable version of it does not, which is the actual gap this section closes.

The mechanism that makes faithfulness testable is claim decomposition. You don't score an entire answer as one grounded-or-not unit, because a single answer routinely mixes a grounded sentence with an invented one, and scoring the whole thing at once hides exactly which part failed. Instead, split the answer into atomic, independently-checkable claims (no pronouns pointing at an earlier claim, no compound "and" statements), then check each claim against the retrieved chunks individually. Here's the decomposition and per-claim entailment logic, written against a generic llm_call so it works with whichever model client you're already using:

"""
Claim decomposition and per-claim entailment checking for RAG faithfulness
testing.

Vendor neutral: the LLM call itself is injected as a callable (llm_call), so
this module works with OpenAI, Anthropic, a local model, or a scripted stub
in tests. It has no hard dependency on any specific provider SDK.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable, List, Optional, Tuple

LLMCall = Callable[[str], str]

DECOMPOSE_PROMPT = """Break the following answer into a numbered list of atomic,
independently checkable factual claims. Each claim must stand on its own: no
pronouns referring to earlier claims, no compound "and" statements.

Answer:
{answer}

Return one claim per line, numbered starting at 1. No other text."""

ENTAILMENT_PROMPT = """Context chunks:
{context}

Claim:
{claim}

Does the context above entail (fully support) this claim? Answer with exactly
one word on the first line: ENTAILED, CONTRADICTED, or UNSUPPORTED. On the
second line, name the chunk number that supports or contradicts it, or NONE
if UNSUPPORTED."""


@dataclass
class ClaimVerdict:
    claim: str
    verdict: str  # "ENTAILED" | "CONTRADICTED" | "UNSUPPORTED"
    supporting_chunk: Optional[str]


def decompose_claims(answer: str, llm_call: LLMCall) -> List[str]:
    """Split an answer into atomic, independently checkable claims."""
    raw = llm_call(DECOMPOSE_PROMPT.format(answer=answer))
    claims: List[str] = []
    for line in raw.strip().splitlines():
        line = line.strip()
        if not line:
            continue
        if "." in line[:3]:
            parts = line.split(".", 1)
        elif ")" in line[:3]:
            parts = line.split(")", 1)
        else:
            parts = [line]
        claims.append(parts[1].strip() if len(parts) == 2 else line)
    return claims


def check_claim_entailment(claim: str, context_chunks: List[str], llm_call: LLMCall) -> ClaimVerdict:
    """Check whether a single atomic claim is entailed by the retrieved context."""
    numbered_context = "\n".join(f"[{i + 1}] {c}" for i, c in enumerate(context_chunks))
    raw = llm_call(ENTAILMENT_PROMPT.format(context=numbered_context, claim=claim))
    lines = [l.strip() for l in raw.strip().splitlines() if l.strip()]
    verdict = lines[0].upper() if lines else "UNSUPPORTED"
    chunk = lines[1] if len(lines) > 1 else None
    return ClaimVerdict(claim=claim, verdict=verdict, supporting_chunk=chunk)


def faithfulness_score(
    answer: str, context_chunks: List[str], llm_call: LLMCall
) -> Tuple[float, List[ClaimVerdict]]:
    """
    Decompose the answer into claims, check each against retrieved context,
    and return (fraction_entailed, per_claim_verdicts). Gate on the fraction
    entailed, not on every claim passing, since one invented detail in an
    otherwise-grounded answer is a real but partial failure.
    """
    claims = decompose_claims(answer, llm_call)
    if not claims:
        return 1.0, []
    verdicts = [check_claim_entailment(c, context_chunks, llm_call) for c in claims]
    entailed = sum(1 for v in verdicts if v.verdict == "ENTAILED")
    return entailed / len(verdicts), verdicts

Claim decomposition turns "is this answer faithful" into a series of small, checkable questions instead of one fuzzy judgment call.

Here's the assertion that ties it together, decomposing an answer, checking each claim against the retrieved context, and gating on the fraction entailed rather than requiring every claim to pass:

"""
The core RAG faithfulness assertion: decompose the answer into atomic
claims, check each one against the retrieved context, and assert on the
fraction entailed, not on exact string match or on every claim passing.

This uses a scripted stub LLM so the test is deterministic and runs with no
network access and no API key. Swap stub_llm for a real client call in your
own suite; lib/faithfulness.py's functions do not change.
"""
from lib.faithfulness import faithfulness_score

RETRIEVED_CONTEXT = [
    "Our refund policy allows returns within 30 days of purchase with a receipt.",
    "Refunds are issued to the original payment method within 5-7 business days.",
]

FAITHFULNESS_THRESHOLD = 0.85  # margin, not exact equality


def make_stub_llm(decompose_response: str, entailment_responses):
    """A scripted fake LLM: the first call returns the decomposition,
    subsequent calls return entailment verdicts in order."""
    responses = iter([decompose_response, *entailment_responses])

    def _call(prompt: str) -> str:
        return next(responses)

    return _call


def test_faithful_answer_passes():
    answer = "You can return items within 30 days, and refunds go back to your original card."
    stub_llm = make_stub_llm(
        decompose_response=(
            "1. Returns are allowed within 30 days.\n"
            "2. Refunds go to the original payment method."
        ),
        entailment_responses=["ENTAILED\n[1]", "ENTAILED\n[2]"],
    )

    score, verdicts = faithfulness_score(answer, RETRIEVED_CONTEXT, stub_llm)

    assert score == 1.0
    assert all(v.verdict == "ENTAILED" for v in verdicts)


def test_hallucinated_claim_fails():
    # The answer invents a detail (a $10 restocking fee) that is not
    # anywhere in the retrieved context. This is the same claim the support
    # bot in the article's opening incident invented.
    answer = "Returns are allowed within 30 days, and there is a $10 restocking fee."
    stub_llm = make_stub_llm(
        decompose_response=(
            "1. Returns are allowed within 30 days.\n"
            "2. There is a $10 restocking fee."
        ),
        entailment_responses=["ENTAILED\n[1]", "UNSUPPORTED\nNONE"],
    )

    score, verdicts = faithfulness_score(answer, RETRIEVED_CONTEXT, stub_llm)

    assert score == 0.5
    unsupported = [v for v in verdicts if v.verdict == "UNSUPPORTED"]
    assert len(unsupported) == 1
    assert "restocking fee" in unsupported[0].claim.lower()


def test_faithfulness_gate_with_margin():
    # In CI you gate on a threshold with margin, not exact equality, because
    # the entailment call itself is a sampled measurement.
    answer = "Returns are allowed within 30 days, and refunds go to your card."
    stub_llm = make_stub_llm(
        decompose_response=(
            "1. Returns are allowed within 30 days.\n"
            "2. Refunds go to your card."
        ),
        entailment_responses=["ENTAILED\n[1]", "ENTAILED\n[2]"],
    )
    score, _ = faithfulness_score(answer, RETRIEVED_CONTEXT, stub_llm)

    assert score >= FAITHFULNESS_THRESHOLD

You don't have to hand-roll this. DeepEval ships a FaithfulnessMetric that does the same claim-level entailment check behind a single call, and the config surface (a threshold, the retrieved context, the actual output) maps directly onto the pieces above:

"""
The same faithfulness assertion expressed with DeepEval's built-in metric,
so you can see the real tool idiom next to the hand-rolled version in
lib/faithfulness.py.

Requires the deepeval package and a configured model provider (DeepEval
calls a real LLM to grade faithfulness by default). This file is provided
as the tool-idiom reference the article names; running it for real needs
deepeval installed and a model configured, unlike the other tests in this
repo, which run fully offline against stub callables.
"""
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase


def test_faithfulness_with_deepeval():
    test_case = LLMTestCase(
        input="What is your refund window?",
        actual_output="You can return items within 30 days, minus a $10 restocking fee.",
        retrieval_context=[
            "Our refund policy allows returns within 30 days of purchase with a receipt.",
            "Refunds are issued to the original payment method within 5-7 business days.",
        ],
    )

    faithfulness = FaithfulnessMetric(
        threshold=0.85,  # margin, not exact equality
        include_reason=True,
    )

    assert_test(test_case, [faithfulness])

Ragas does the same thing at the dataset level rather than one test case at a time, which is the shape you want once you're scoring a batch of question-answer-context triples instead of asserting on a single example:

"""
The same faithfulness assertion expressed with Ragas, evaluated over a small
dataset instead of a single test case. Ragas scores the whole dataset at
once and returns a per-row faithfulness column.

Requires the ragas and datasets packages and a configured model provider
(Ragas calls a real LLM to grade faithfulness by default). This file is
provided as the tool-idiom reference the article names; running it for real
needs those libraries installed and a model configured, unlike the other
tests in this repo, which run fully offline against stub callables.
"""
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness

FAITHFULNESS_THRESHOLD = 0.85  # margin, not exact equality


def test_faithfulness_with_ragas():
    dataset = Dataset.from_dict(
        {
            "question": ["What is your refund window?"],
            "answer": ["You can return items within 30 days, minus a $10 restocking fee."],
            "contexts": [
                [
                    "Our refund policy allows returns within 30 days of purchase with a receipt.",
                    "Refunds are issued to the original payment method within 5-7 business days.",
                ]
            ],
        }
    )

    result = evaluate(dataset, metrics=[faithfulness])
    score = result["faithfulness"][0]

    assert score < FAITHFULNESS_THRESHOLD, (
        "This answer invents a restocking fee not present in the retrieved "
        "context; Ragas should score it below the faithfulness bar."
    )

All three versions, the hand-rolled one and both library equivalents, are computing the same thing: what fraction of the atomic claims in this answer are actually supported by what the model was given. If your RAG pipeline's retrieval step itself is the thing under suspicion (wrong chunk, not wrong claim), how to test a RAG pipeline goes one layer deeper, into testing retrieval quality separately from generation quality.

How Autonoma Closes the Gap Faithfulness Scores Can't See

Every check above, hand-rolled or via DeepEval or Ragas, scores the same thing: the text of the answer against the text of the retrieved context. That's the right thing to score, and it's still only half the failure mode that actually reaches users. A faithfulness score can come back clean, every claim entailed, threshold cleared, and the application the AI feature is wired into can still do the wrong thing with that grounded answer: prefill the wrong field, route the user to the wrong screen, trigger a side effect the grounded text never implied. The score tells you the words were honest. It has no visibility into what happened after the words were generated.

That's the layer Autonoma operates in, and it's deliberately not another eval framework: it never scores a faithfulness metric or grades a model's output. Autonoma's Planner reads your codebase, the routes, components, and flows a genAI feature actually touches, and plans behavioral end-to-end test cases around what should happen inside the running application after that feature responds, including generating the database state each scenario needs. Executor drives those tests against the real UI in a live preview environment, the same environment a deploy would use, so the assertion is "did the app end up in the right state," not "was the text grounded." Reviewer classifies what it finds, a genuine bug in how the feature's output got wired into the app, an agent error, or a plan-to-code mismatch, and Diffs Agent keeps that coverage current on every pull request by reading the code diff instead of letting the suite quietly rot. A hallucination test tells you the claim wasn't supported. A behavioral test tells you whether the unsupported claim still made it into a database row or a rendered screen, which is the outcome that actually reaches a user.

LLM-as-Judge for Factuality

Once you're past exact-match and semantic similarity, most factuality checks end up leaning on a judge model to read the answer and the context and decide whether the claims hold up. The failure mode here isn't the concept, it's the rubric. A prompt that asks "is this answer accurate, rate 1 to 5" produces a number that drifts run to run and means something slightly different every time you use it, because "accurate" was never defined as a criterion. A rubric that forces a binary entailment verdict per claim, and forces the judge to cite which retrieved chunk supports (or fails to support) that verdict, produces something you can actually threshold and trust.

"""
An LLM-as-judge rubric for factuality that avoids the failure mode of a vague
"is this accurate, rate 1 to 5" score. It forces a binary entailment verdict
per atomic claim and a citation of the chunk that supports (or fails to
support) it, the same discipline lib/faithfulness.py's ENTAILMENT_PROMPT
uses.

This module wraps that pattern into a single "judge the whole answer" call,
useful when you want one aggregate factuality verdict rather than per-claim
verdicts computed one entailment call at a time.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable, List

LLMCall = Callable[[str], str]

FACTUALITY_RUBRIC_PROMPT = """You are grading an AI-generated answer for factual
accuracy against the context it was given. Do not use your own knowledge, use
ONLY the context chunks below.

Context chunks:
{context}

Answer to grade:
{answer}

For EACH factual claim in the answer, output one line in this exact format:
CLAIM: <the claim> | VERDICT: <ENTAILED or UNSUPPORTED or CONTRADICTED> | CHUNK: <chunk number or NONE>

Do not give an overall 1-5 score. Do not skip a claim because it seems
obviously true. If it is not stated in the context chunks, it is UNSUPPORTED
even if it sounds correct."""


@dataclass
class JudgeLine:
    claim: str
    verdict: str
    chunk: str


def parse_judge_output(raw: str) -> List[JudgeLine]:
    lines: List[JudgeLine] = []
    for row in raw.strip().splitlines():
        row = row.strip()
        if "VERDICT:" not in row or "CHUNK:" not in row:
            continue
        claim_part, rest = row.split("VERDICT:", 1)
        verdict_part, chunk_part = rest.split("CHUNK:", 1)
        lines.append(
            JudgeLine(
                claim=claim_part.replace("CLAIM:", "").strip(" |"),
                verdict=verdict_part.strip(" |"),
                chunk=chunk_part.strip(),
            )
        )
    return lines


def judge_factuality(context_chunks: List[str], answer: str, llm_call: LLMCall) -> List[JudgeLine]:
    """
    Run the factuality rubric once against the whole answer and return the
    per-claim verdicts the judge produced, each with a cited chunk.
    """
    numbered_context = "\n".join(f"[{i + 1}] {c}" for i, c in enumerate(context_chunks))
    raw = llm_call(FACTUALITY_RUBRIC_PROMPT.format(context=numbered_context, answer=answer))
    return parse_judge_output(raw)

Here's the assertion built on that rubric, gating on the fraction of claims the judge marks entailed rather than trusting a single overall verdict:

"""
Using the rubric from lib/factuality_judge.py to gate on the fraction of
claims marked ENTAILED, plus a note on the rubric's real limits: the judge
itself is non-deterministic, it is biased toward confident and verbose
phrasing, and it needs its own calibration against a small set of
human-labeled examples before you trust its threshold in CI.

Runs fully offline against a scripted stub judge, no network access and no
API key required.
"""
from lib.factuality_judge import judge_factuality

CONTEXT = [
    "Our refund policy allows returns within 30 days of purchase with a receipt.",
    "Refunds are issued to the original payment method within 5-7 business days.",
]


def stub_judge(prompt: str) -> str:
    return (
        "CLAIM: Returns are allowed within 30 days | VERDICT: ENTAILED | CHUNK: 1\n"
        "CLAIM: There is a $10 restocking fee | VERDICT: UNSUPPORTED | CHUNK: NONE"
    )


def test_judge_flags_unsupported_claim():
    answer = "Returns are allowed within 30 days, and there is a $10 restocking fee."
    verdicts = judge_factuality(CONTEXT, answer, stub_judge)

    entailed = [v for v in verdicts if v.verdict == "ENTAILED"]
    unsupported = [v for v in verdicts if v.verdict == "UNSUPPORTED"]

    assert len(entailed) == 1
    assert len(unsupported) == 1
    assert "restocking fee" in unsupported[0].claim.lower()


def test_judge_score_gates_on_fraction_not_single_run():
    # A single judge call is a sampled measurement, not ground truth. Treat
    # this like any other flaky-prone check: gate on a fraction across
    # multiple runs (see lib/repeat_and_threshold.py), not a single
    # pass/fail from one call.
    answer = "Returns are allowed within 30 days, and there is a $10 restocking fee."
    verdicts = judge_factuality(CONTEXT, answer, stub_judge)
    fraction_entailed = sum(1 for v in verdicts if v.verdict == "ENTAILED") / len(verdicts)
    assert fraction_entailed < 0.85  # below threshold: this answer should fail

Where Does the Judge Get It Wrong?

A judge with a good rubric is still a probabilistic system judging another probabilistic system, and pretending otherwise is how a factuality suite quietly becomes untrustworthy. The judge call is itself non-deterministic: run the same claim through it twice and you can get two different verdicts, especially near the entailment boundary. It carries known biases, toward longer and more confident-sounding answers, and toward outputs that resemble its own training distribution, which matters most when your judge and your generator are the same model family and share the same blind spots. A judge tuned on someone else's rubric needs its own calibration against a small set of human-labeled examples before you trust its threshold in CI, the same way you'd never trust a new hire's bug triage without checking their first few calls against a senior engineer's. None of that makes LLM-as-judge unusable. It makes it a component you calibrate and monitor, not a ground truth you accept on the first output.

Why Does the Same Faithfulness Check Pass One Run and Fail the Next?

A faithfulness or judge score is a sampled measurement, not a fixed property of the answer, because both the generator and the judge are non-deterministic. Gating on a single run conflates three genuinely different problems: the assertion threshold is too strict for normal variance, the prompt is ambiguous enough that the model reasonably answers two different ways, or the retrieval is actually wrong and something real changed. Treating all three as "the test is flaky" and loosening the threshold until it stops complaining fixes none of them and hides all three.

The fix is the same one that shows up across every check in this cluster: run the scenario N times and gate on a pass rate with margin, not on unanimous agreement. Five runs with four passing is a defensible floor for most factuality checks; a scenario that drops from 5-of-5 to 2-of-5 over a week is a real signal (a prompt edit, a model version bump, retrieved context that quietly got worse), the exact kind of signal a single-run assertion would never surface. For a deeper look at the broader pattern behind this, including semantic-similarity assertions and the general "is my test too strict or is the model actually wrong" decision tree, see how to test non-deterministic AI outputs.

"""
A statistical wrapper for non-deterministic scores. A faithfulness or
factuality score is itself a sampled measurement, since both the generator
and, if you use one, the judge are non-deterministic. Gating on a single run
conflates three different problems: the assertion threshold is too strict,
the prompt is too ambiguous, or the retrieval is genuinely wrong.

Run the check N times and gate on a pass rate with margin instead of exact
equality.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable, List


@dataclass
class RepeatedResult:
    scores: List[float]
    pass_rate: float
    mean_score: float


def run_n_times(check_fn: Callable[[], float], n: int, threshold: float) -> RepeatedResult:
    """
    Run check_fn n times (each call should independently re-generate the
    answer and re-score it), and report the pass rate against threshold
    plus the mean score.
    """
    scores = [check_fn() for _ in range(n)]
    passes = sum(1 for s in scores if s >= threshold)
    return RepeatedResult(
        scores=scores,
        pass_rate=passes / n,
        mean_score=sum(scores) / n,
    )


def assert_stable_pass_rate(result: RepeatedResult, min_pass_rate: float = 0.8) -> None:
    """
    Gate on a pass rate, not unanimous agreement. A drop from 5-of-5 to
    3-of-5 across a week of runs is a real signal (a prompt edit, a model
    version bump, or retrieval degrading). It is not the same failure as a
    test that has always been too strict to ever pass.
    """
    assert result.pass_rate >= min_pass_rate, (
        f"Pass rate {result.pass_rate:.0%} (mean score {result.mean_score:.2f}) "
        f"fell below the {min_pass_rate:.0%} floor across repeated runs. "
        "Before loosening the threshold, check whether retrieval degraded "
        "first: a wrong chunk explains a real failure, a flaky judge does not."
    )

Building a Hallucination Regression Set From Real Incidents

Every check above is aimed at catching a hallucination before it ships. The most durable one is aimed at making sure the same hallucination never ships twice. Every hallucination that reaches a real user is a gift, in the specific sense that it's a concrete, verified failure mode you now know how to describe precisely: the question, the context that was retrieved, the exact wrong claim, and why it happened. Turn that into a permanent test case, and a recurrence of that specific failure fails the build instead of reaching a user a second time.

[
  {
    "incident_id": "INC-2026-0114",
    "question": "What is your refund window?",
    "retrieved_context": [
      "Our refund policy allows returns within 30 days of purchase with a receipt.",
      "Refunds are issued to the original payment method within 5-7 business days."
    ],
    "hallucinated_answer": "You can return items within 30 days, minus a $10 restocking fee.",
    "why_it_reached_production": "The model added a plausible-sounding fee that appears nowhere in the retrieved policy text. The response was fluent, well-formatted, and confidently wrong, and it passed every conventional test the team had at the time.",
    "grounded_claims": [
      "Returns are allowed within 30 days of purchase with a receipt.",
      "Refunds are issued to the original payment method within 5-7 business days."
    ]
  },
  {
    "incident_id": "INC-2026-0142",
    "question": "Can I get a refund on a discounted item?",
    "retrieved_context": [
      "Our refund policy allows returns within 30 days of purchase with a receipt.",
      "Discounted and clearance items follow the same 30-day refund policy as full-price items."
    ],
    "hallucinated_answer": "Discounted items are final sale and cannot be refunded.",
    "why_it_reached_production": "The model answered from a common but stale assumption about clearance items instead of the retrieved chunk that explicitly overrides it for this store.",
    "grounded_claims": [
      "Discounted items follow the same 30-day refund policy as full-price items."
    ]
  },
  {
    "incident_id": "INC-2026-0171",
    "question": "Do you offer price matching?",
    "retrieved_context": [
      "We do not offer price matching against competitor prices.",
      "We do offer a one-time price adjustment if the same item drops in price on our own store within 14 days of your purchase."
    ],
    "hallucinated_answer": "Yes, we price match any competitor's listed price.",
    "why_it_reached_production": "The model conflated the internal 14-day price-adjustment policy with external competitor price matching, which the retrieved context explicitly rules out.",
    "grounded_claims": [
      "We do not price match competitors.",
      "We offer a one-time price adjustment within 14 days for our own price drops."
    ]
  }
]

"""
Every hallucination that reached a real user becomes a permanent regression
case here. A recurrence of any of these exact failure patterns fails CI,
which is the only thing that reliably stops the same hallucination from
shipping twice.

Runs fully offline: each incident is replayed against a scripted stub LLM
built from that incident's own fixture data, no network access required.
"""
import json
import pathlib

import pytest

from lib.faithfulness import faithfulness_score

FIXTURES_PATH = pathlib.Path(__file__).parent.parent / "fixtures" / "hallucination_incidents.json"
INCIDENTS = json.loads(FIXTURES_PATH.read_text())

FAITHFULNESS_THRESHOLD = 0.85  # margin, not exact equality


def make_stub_llm_for_incident(incident: dict):
    """
    Deterministic stand-in for the real generate-then-judge pipeline,
    scripted from the incident's own hallucinated answer, so this file runs
    in CI with no API key. Swap stub_llm for your actual pipeline once this
    incident is wired into your live regression harness: the point of this
    fixture is the assertion shape, not the stub.
    """

    def _call(prompt: str) -> str:
        if "atomic, independently checkable" in prompt:
            return f"1. {incident['hallucinated_answer']}"
        # The hallucinated claim, by definition, is not supported by the
        # retrieved context recorded for this incident.
        return "UNSUPPORTED\nNONE"

    return _call


@pytest.mark.parametrize("incident", INCIDENTS, ids=[i["incident_id"] for i in INCIDENTS])
def test_past_hallucination_does_not_recur(incident):
    stub_llm = make_stub_llm_for_incident(incident)
    score, verdicts = faithfulness_score(
        incident["hallucinated_answer"],
        incident["retrieved_context"],
        stub_llm,
    )

    assert score < FAITHFULNESS_THRESHOLD, (
        f"{incident['incident_id']} recurred: an answer this close to the "
        f"original hallucination is passing the faithfulness gate again. "
        f"Root cause on file: {incident['why_it_reached_production']}"
    )

None of this protects anything if it only runs on a laptop before a release someone remembers to trigger manually. Wire it into CI so a recurrence blocks the merge automatically, the same way any other regression suite does:

name: Hallucination Regression Suite

on:
  pull_request:
  push:
    branches: [main]

jobs:
  faithfulness-and-regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install offline test dependencies
        run: pip install pytest

      - name: Run faithfulness assertions
        run: pytest tests/test_rag_faithfulness.py tests/test_reference_based_check.py tests/test_reference_free_groundedness.py tests/test_llm_judge_factuality.py -v

      - name: Run the hallucination regression set
        run: pytest tests/test_hallucination_regression.py -v

      - name: Fail the build on any recurrence
        if: failure()
        run: |
          echo "::error::A past production hallucination has recurred. See the failing incident ID above."
          exit 1

A regression set built this way compounds in a way a static eval set never does. Every incident makes the next release a little harder to break the same way twice, and after a few months of real production traffic feeding it, the regression set becomes a more accurate map of your actual failure modes than any golden set someone wrote up front ever could be.

Catching the Next One Before It Ships

Stack these checks in the order they earn their keep. Reference-based checks are cheap and precise but only exist where someone labeled an answer, so they belong on your highest-stakes, curated scenarios. Reference-free and faithfulness checks are what actually runs on everything else, since they need no ground truth, only the context the model was given. LLM-as-judge fills in wherever a rule-based check can't reach, provided the rubric forces per-claim citations instead of a vague overall score, and provided you never forget the judge itself needs calibration. Run all of it N times, not once, so a threshold gate reflects an actual pattern instead of a single lucky or unlucky sample. Then feed every real production miss back into a regression set that makes the specific failure impossible to ship a second time.

Faithfulness and factuality checks answer whether the words were honest. For the rarer but sharper case, a grounded, honest-sounding answer that still drove the application to do the wrong thing, that's the layer Autonoma's behavioral testing covers, above and separate from anything a hallucination score can measure.

Frequently Asked Questions

Run two complementary checks: a reference-based check that compares the answer to a labeled, known-good answer using semantic similarity, and a reference-free check that verifies the answer is grounded in whatever context the model was actually given. For RAG systems, decompose the answer into atomic claims and check each one for entailment against the retrieved chunks, gating on the fraction supported rather than requiring every claim to pass.

DeepEval and Ragas are the two most commonly used, and both implement claim-level faithfulness scoring behind a single metric call rather than requiring you to hand-roll the decomposition and entailment logic yourself. The choice of tool matters less than the rubric: a claim-by-claim entailment check with forced chunk citation catches far more than a vague 'is this accurate, 1 to 5' judge prompt.

Test faithfulness (does the answer stay grounded in the retrieved chunks) separately from retrieval quality (did the retriever pull back the right chunks in the first place). A faithful answer built on the wrong retrieved chunk is still a real production failure, just a different one, which is why testing a RAG pipeline needs both a faithfulness check on the generation step and a separate check on what the retriever returned.

Run reference-free faithfulness and groundedness checks continuously in CI, since they need no labeled ground truth and can run against realistic inputs before a release ships. Repeat non-deterministic checks multiple times and gate on a pass rate instead of a single run, and turn every hallucination that does slip through into a permanent regression test case so the same failure can't reach production a second time.

It's reliable when the rubric forces a binary entailment verdict per atomic claim with a cited supporting chunk, and unreliable when it asks for a vague overall accuracy score. Even with a good rubric, the judge itself is non-deterministic and biased toward verbose, confident-sounding answers, so it needs its own calibration against human-labeled examples and should be run multiple times with a pass-rate threshold rather than trusted on a single call.

How to Test for AI Hallucinations: A Code-First Guide

What Counts as a Hallucination (and What Doesn't)

Reference-Based vs Reference-Free Checks

RAG Faithfulness: Keeping the Answer Grounded in Retrieved Context

How Autonoma Closes the Gap Faithfulness Scores Can't See

LLM-as-Judge for Factuality

Where Does the Judge Get It Wrong?

Why Does the Same Faithfulness Check Pass One Run and Fail the Next?

Building a Hallucination Regression Set From Real Incidents

Catching the Next One Before It Ships

Frequently Asked Questions

How do you test for AI hallucinations?

What are the best hallucination testing tools?

How do you test RAG hallucinations specifically?

How do you catch AI hallucinations before they reach production?

Is LLM-as-judge reliable for detecting hallucinations?

How to Test for AI Hallucinations: A Code-First Guide

What Counts as a Hallucination (and What Doesn't)

Reference-Based vs Reference-Free Checks

RAG Faithfulness: Keeping the Answer Grounded in Retrieved Context

How Autonoma Closes the Gap Faithfulness Scores Can't See

LLM-as-Judge for Factuality

Where Does the Judge Get It Wrong?

Why Does the Same Faithfulness Check Pass One Run and Fail the Next?

Building a Hallucination Regression Set From Real Incidents

Catching the Next One Before It Ships

Frequently Asked Questions

How do you test for AI hallucinations?

What are the best hallucination testing tools?

How do you test RAG hallucinations specifically?

How do you catch AI hallucinations before they reach production?

Is LLM-as-judge reliable for detecting hallucinations?

Related articles

How to Test AI Agents That Take Actions (Tool Calls)

Chatbot Automation Testing: Why Assertions Fail

Ghost Inspector Alternative: Recorder, Framework, or AI?

How to Test the Auth Code an AI Agent Wrote