LLM evals are automated checks that score a model's output against a quality bar, correctness, faithfulness, tone, using a scoring function instead of an exact-match assertion, which is what separates them from a normal unit test. Running evals in CI/CD means splitting the pipeline into two lanes: fast, free, deterministic tests that gate every commit, and slower, token-costly eval tests that run on merge to main and nightly, scored by averaging several runs against a tolerance threshold instead of failing on any single miss.
Ask a team if they run evals in CI and most say yes. Ask to see the job, and what shows up is one Python script, run by hand before a release, scored by someone eyeballing a spreadsheet of forty examples. Nothing gates a merge. Nothing runs on a schedule. Nobody would notice a five-point accuracy drop until a customer complained about it.
That gap, between "we run evals" and an actual CI gate, is the subject of this piece. What a deterministic test proves that an eval test cannot. Why the two need entirely different scheduling inside your pipeline. And how to wire a threshold gate that survives the fact that the same prompt against the same model can return a different string on every single call.
The first time I wired an eval suite into a merge queue, I made the obvious mistake: I ran the whole thing on every commit, the same way the rest of the test suite ran. It worked, for about a week, until a teammate opened a PR with six small commits pushed in an afternoon and the eval job burned through a token bill that would have paid for a month of the model API on its own. The fix wasn't a smarter model or a cheaper provider. It was admitting that an eval test and a unit test are not the same kind of check, and treating them like the same thing is what makes the bill hurt.
What an LLM Eval Actually Tests (and Why a Unit Test Can't Do the Job)
A deterministic test in your existing suite asserts a fact about code: given this input, the function returns exactly this output, every time. An LLM eval asserts something softer: given this input, is the model's answer good enough, correct, grounded in the source material, on topic, appropriately toned, according to a scoring function rather than a string comparison. The two are catching different failure modes, and that's the whole reason they need to live in different parts of your pipeline.
An exact-match assertion can't do an eval's job because the same prompt against the same model rarely returns the same string twice. "The refund was processed" and "I've gone ahead and processed your refund" are both correct answers to the same support query, and a test that fails one of them because the words don't line up is a test your team will start ignoring within a sprint. An eval test replaces the string comparison with a scoring function: an LLM-as-judge call, a semantic-similarity check against a reference answer, a faithfulness check against retrieved context, or a rubric score, and it asserts that the score clears a threshold, not that the text matches a fixture.
DeepEval's own documentation gets closer to this than most of the search results on "how to run evals in CI/CD" do. Its LLMTestCase and assert_test primitives are a legitimate pytest-native way to express exactly this kind of threshold assertion, and its docs are one of the few places that explicitly recommends averaging three or more runs to absorb non-deterministic variance rather than trusting a single pass. Where that documentation stops short is the pipeline wrapped around those primitives: what runs on every commit versus what waits for merge, how the dataset backing those test cases gets versioned, and what the actual workflow file looks like end to end. That's the gap the rest of this piece fills in.
Dimension
Deterministic test
Eval test
What it asserts
Exact output or behavior
Score above a threshold
Speed
Milliseconds
Seconds per case
Cost
Free, no model call
Paid model tokens
Determinism
Same input, same output
Same input, varying output
When it runs
Every commit
Merge to main, nightly
That last row is the spine of everything that follows. A deterministic test costs nothing to run on every commit, so it runs on every commit. An eval test costs a model call, real tokens, real seconds, real dollars, per case, and running the full eval suite on every keystroke would push your CI bill and your CI wait time in the wrong direction at the same time. The scheduling decision below isn't a nice-to-have. It's the difference between a pipeline that stays affordable and one that quietly gets disabled the first time someone notices the invoice.
The Pipeline Pattern: Deterministic on Every Commit, Evals on Merge and Nightly
The pattern that holds up in practice is a two-speed pipeline. Deterministic tests, the ones checking your prompt-building code, your response parsing, your retry handling, your API schema, run on every single commit and every pull request, exactly like any other unit test suite. They're free and they're fast, so there's no reason to gate them any less aggressively than you'd gate a normal test.
Eval tests run on two triggers only: merge to main, and a nightly cron. Merge to main catches a deliberate change, a new prompt, a new model version pinned in config, a new retrieval source, before it reaches production. The nightly run catches the thing a merge trigger cannot: a model provider quietly rolling a new version behind the same API name, with no pull request and no commit for a merge hook to catch. Skip the nightly run and a silent provider-side regression sits undetected until a user notices. Skip the merge-gated run and a bad prompt edit ships straight to production, waiting for the next night's cron to flag it.
Deterministic tests gate every commit. Eval tests gate two things only: a merge, and the clock.
Worth being precise about what each stage actually proves. The eval job above scores whether the model's response clears a quality bar. It says nothing about whether that response, once it reaches your product, actually flips the right database row or renders the right screen, a gap Autonoma closes with a separate behavioral E2E layer that can run as its own gate further down the same pipeline.
The GitHub Actions Workflow, Wired End to End
Here's what that two-speed pattern looks like as an actual pipeline, not a diagram. Two jobs, two triggers: deterministic-tests runs on every push and pull request, and llm-evals runs on push to main plus a nightly schedule: cron, with a workflow_dispatch trigger so anyone can kick it off by hand before a release.
name: evals# Two jobs, two different economics.## deterministic-tests every push and PR. No model calls, no key, seconds.# llm-evals merge to main, nightly at 02:00 UTC, or on demand.# Real model calls, real money, minutes.## Putting model-backed evals on every push is how teams end up disabling them.# Putting them nowhere is how drift ships. This split is the compromise.on: push: branches: ["**"] pull_request: schedule: # 02:00 UTC nightly. Catches drift from vendor-side model updates that no # commit of yours would ever trigger. - cron: '0 2 * * *' workflow_dispatch:permissions: contents: readconcurrency: group: evals-${{ github.workflow }}-${{ github.ref }} cancel-in-progress: ${{ github.event_name == 'pull_request' }}jobs: deterministic-tests: name: Deterministic tests (no model calls) if: github.event_name == 'push' || github.event_name == 'pull_request' runs-on: ubuntu-latest timeout-minutes: 10 steps: - name: Check out the repository uses: actions/checkout@v7 - name: Set up Python 3.11 uses: actions/setup-python@v7 with: python-version: '3.11' cache: pip cache-dependency-path: requirements.txt - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Run deterministic tests run: pytest tests/test_deterministic.py -v llm-evals: name: LLM evals (model calls) if: >- (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' runs-on: ubuntu-latest timeout-minutes: 30 env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} steps: - name: Check out the repository uses: actions/checkout@v7 - name: Set up Python 3.11 uses: actions/setup-python@v7 with: python-version: '3.11' cache: pip cache-dependency-path: requirements.txt - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt # This public reference repo has no API key, so it warns and skips rather # than showing a permanently red badge to readers. # # >>> IN YOUR OWN REPO, MAKE THIS A HARD FAILURE. <<< # Replace the two `have_key=false` lines below with `exit 1`. An eval job # that skips itself because a secret went missing is indistinguishable # from an eval job that passed, and that is precisely the failure mode # this whole pipeline exists to prevent. - name: Check for the API key id: preflight run: | if [ -z "${OPENAI_API_KEY}" ]; then echo "::warning::OPENAI_API_KEY is not set, so the eval run was skipped. Add it under Settings > Secrets and variables > Actions." echo "have_key=false" >> "$GITHUB_OUTPUT" else echo "have_key=true" >> "$GITHUB_OUTPUT" fi - name: Run eval suite id: eval_suite if: steps.preflight.outputs.have_key == 'true' # Keep going on failure so the scoring step can still publish a summary # explaining which case regressed. The gate below is what fails the job. continue-on-error: true run: | mkdir -p reports pytest tests/test_evals.py -v \ --json-report \ --json-report-file=reports/eval-report.json - name: Upload raw eval report if: always() && steps.preflight.outputs.have_key == 'true' uses: actions/upload-artifact@v7 with: name: eval-report-${{ github.run_id }} path: reports/eval-report.json if-no-files-found: warn retention-days: 30 - name: Score evals and gate on regression if: steps.preflight.outputs.have_key == 'true' run: | set +e python scripts/run_evals.py \ --reuse-report \ --report reports/eval-report.json \ --summary-out eval-summary.md status=$? set -e if [ -f eval-summary.md ]; then cat eval-summary.md >> "$GITHUB_STEP_SUMMARY" else echo "## LLM eval results" >> "$GITHUB_STEP_SUMMARY" echo "" >> "$GITHUB_STEP_SUMMARY" echo "The scoring step produced no summary. Check the job logs." >> "$GITHUB_STEP_SUMMARY" fi exit $status
The eval job's last step is a reporting script, not the raw pytest output. Pytest tells you which assertions failed. It doesn't summarize the run for a teammate skimming a failed check three days later, and it won't exit with a clean non-zero code when the aggregate score has regressed even if every individual case still limps over its own threshold. Here's that script: it runs the suite, computes the averaged score per case, compares each average against a stored baseline file, and fails the job the moment any case regresses past its tolerance.
#!/usr/bin/env python3"""Run the eval suite, compare against the baseline, and gate CI on regression.This is the last step of the eval job. `pytest` already knows whether each casepassed, but a red X on a test name is a bad CI artifact for evals: you want tosee every case, its averaged score, its baseline, and the delta, ranked worstfirst, so you can tell "the model drifted two points" apart from "the refundprompt is broken".Two modes:* `python scripts/run_evals.py` Runs pytest itself, then scores. This is the local one-liner.* `python scripts/run_evals.py --reuse-report --report reports/eval-report.json` Scores a JSON report a previous pytest step already produced. This is what CI uses, so the eval job pays for exactly one set of model calls.Exit code is 1 if any case regressed past tolerance or produced no score at all,and 0 otherwise. That exit code is the CI gate."""from __future__ import annotationsimport argparseimport jsonimport subprocessimport sysfrom pathlib import Pathfrom typing import Any, Dict, List, OptionalROOT = Path(__file__).resolve().parents[1]DEFAULT_REPORT = ROOT / "reports" / "eval-report.json"DEFAULT_TESTS = "tests/test_evals.py"BASELINE_PATH = ROOT / "data" / "baseline_scores.json"STATUS_PASS = "pass"STATUS_REGRESSED = "regressed"STATUS_NO_SCORE = "no-score"STATUS_ICON = { STATUS_PASS: "PASS", STATUS_REGRESSED: "REGRESSED", STATUS_NO_SCORE: "NO SCORE",}def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace: parser = argparse.ArgumentParser( description="Run LLM evals and fail on score regression.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) parser.add_argument( "--report", type=Path, default=DEFAULT_REPORT, help="Path to the pytest JSON report to write or read.", ) parser.add_argument( "--reuse-report", action="store_true", help="Score an existing report instead of running pytest again.", ) parser.add_argument( "--tests", default=DEFAULT_TESTS, help="Test path passed to pytest when running the suite.", ) parser.add_argument( "--tolerance", type=float, default=None, help="Override the tolerance from baseline_scores.json.", ) parser.add_argument( "--summary-out", type=Path, default=None, help="Also write the markdown summary to this file.", ) parser.add_argument( "--write-baseline", action="store_true", help="Record this run's averages as the new baseline instead of gating.", ) return parser.parse_args(argv)def load_baseline(path: Path = BASELINE_PATH) -> Dict[str, Any]: if not path.exists(): raise FileNotFoundError(f"baseline scores not found at {path}") with path.open("r", encoding="utf-8") as handle: return json.load(handle)def run_pytest(report_path: Path, tests: str) -> int: """Run the eval suite with a JSON report. Returns pytest's exit code.""" report_path.parent.mkdir(parents=True, exist_ok=True) command = [ sys.executable, "-m", "pytest", tests, "-v", "--json-report", f"--json-report-file={report_path}", ] print(f"$ {' '.join(command)}\n", flush=True) completed = subprocess.run(command, cwd=ROOT) return completed.returncodedef load_report(path: Path) -> Dict[str, Any]: if not path.exists(): raise FileNotFoundError( f"pytest JSON report not found at {path}. " f"Install pytest-json-report (pip install -r requirements.txt) or drop " f"--reuse-report so this script runs the suite itself." ) with path.open("r", encoding="utf-8") as handle: return json.load(handle)def extract_rows(report: Dict[str, Any], tolerance: Optional[float]) -> List[Dict[str, Any]]: """Turn the pytest JSON report into one row per eval case.""" rows: List[Dict[str, Any]] = [] for test in report.get("tests", []): metadata = test.get("metadata") or {} eval_data = metadata.get("eval") node_id = test.get("nodeid", "<unknown>") if not eval_data: # Skipped (no API key), errored during collection, or crashed before # it recorded a score. Either way there is nothing to compare, and a # silently missing eval must not read as a pass. rows.append( { "case_id": node_id.split("[", 1)[-1].rstrip("]") if "[" in node_id else node_id, "runs": 0, "average": None, "baseline": None, "delta": None, "threshold": None, "grounded": False, "status": STATUS_NO_SCORE, "detail": test.get("outcome", "unknown"), } ) continue effective_tolerance = tolerance if tolerance is not None else eval_data["tolerance"] baseline = float(eval_data["baseline"]) average = float(eval_data["average"]) threshold = round(baseline - effective_tolerance, 4) regressed = average < threshold rows.append( { "case_id": eval_data["case_id"], "runs": eval_data["runs"], "average": average, "baseline": baseline, "delta": round(average - baseline, 4), "threshold": threshold, "grounded": bool(eval_data.get("grounded")), "status": STATUS_REGRESSED if regressed else STATUS_PASS, "detail": eval_data.get("worst_response", "") if regressed else "", "scores": eval_data.get("scores", []), } ) # Worst first: the thing that broke should be the first thing you read. rows.sort(key=lambda row: (row["delta"] if row["delta"] is not None else -99)) return rowsdef format_cell(value: Optional[float], places: int = 4) -> str: return "-" if value is None else f"{value:.{places}f}"def build_summary(rows: List[Dict[str, Any]], tolerance: float) -> str: """Render the markdown summary written to stdout and $GITHUB_STEP_SUMMARY.""" regressed = [row for row in rows if row["status"] == STATUS_REGRESSED] missing = [row for row in rows if row["status"] == STATUS_NO_SCORE] scored = [row for row in rows if row["average"] is not None] lines: List[str] = ["## LLM eval results", ""] if not rows: lines.append("No eval cases ran. Check that `tests/test_evals.py` collected anything.") return "\n".join(lines) + "\n" if regressed or missing: lines.append( f"**{len(regressed)} regressed, {len(missing)} without a score, " f"{len(scored) - len(regressed)} passing** (tolerance {tolerance:.2f})." ) else: lines.append(f"**All {len(scored)} cases within tolerance {tolerance:.2f}.**") lines.append("") lines.append("| Case | Runs | Avg | Baseline | Delta | Threshold | Grounded | Status |") lines.append("| --- | ---: | ---: | ---: | ---: | ---: | :---: | --- |") for row in rows: delta = row["delta"] delta_text = "-" if delta is None else f"{delta:+.4f}" lines.append( "| `{case}` | {runs} | {avg} | {base} | {delta} | {thresh} | {grounded} | {status} |".format( case=row["case_id"], runs=row["runs"] or "-", avg=format_cell(row["average"]), base=format_cell(row["baseline"]), delta=delta_text, thresh=format_cell(row["threshold"]), grounded="yes" if row["grounded"] else "no", status=STATUS_ICON[row["status"]], ) ) lines.append("") if scored: overall = sum(row["average"] for row in scored) / len(scored) lines.append(f"Mean score across {len(scored)} scored cases: **{overall:.4f}**") lines.append("") for row in regressed: lines.append(f"### Regression: `{row['case_id']}`") lines.append("") lines.append( f"Averaged {row['average']:.4f} over {row['runs']} runs, " f"below the threshold {row['threshold']:.4f} " f"(baseline {row['baseline']:.4f} minus tolerance {tolerance:.2f})." ) if row.get("scores"): lines.append("") lines.append(f"Individual run scores: `{row['scores']}`") if row.get("detail"): lines.append("") lines.append("Worst response:") lines.append("") lines.append("```text") lines.append(str(row["detail"])) lines.append("```") lines.append("") for row in missing: lines.append( f"- `{row['case_id']}` produced no score (pytest outcome: {row['detail']}). " f"If this says `skipped`, `OPENAI_API_KEY` was not set." ) if missing: lines.append("") return "\n".join(lines) + "\n"def write_baseline(rows: List[Dict[str, Any]], path: Path = BASELINE_PATH) -> None: """Overwrite the recorded baselines with this run's averages.""" scored = [row for row in rows if row["average"] is not None] if not scored: raise RuntimeError("refusing to write a baseline from a run that scored nothing") baseline = load_baseline(path) baseline["cases"] = {row["case_id"]: row["average"] for row in sorted(scored, key=lambda r: r["case_id"])} baseline["baseline_version"] = int(baseline.get("baseline_version", 0)) + 1 with path.open("w", encoding="utf-8") as handle: json.dump(baseline, handle, indent=2) handle.write("\n") print(f"Wrote {len(scored)} baselines to {path} (version {baseline['baseline_version']}).")def main(argv: Optional[List[str]] = None) -> int: args = parse_args(argv) baseline_config = load_baseline() tolerance = args.tolerance if args.tolerance is not None else float(baseline_config.get("tolerance", 0.05)) if args.reuse_report and args.report.exists(): print(f"Reusing existing pytest report: {args.report}\n", flush=True) else: if args.reuse_report: print( f"--reuse-report was passed but {args.report} does not exist; running pytest.\n", flush=True, ) run_pytest(args.report, args.tests) report = load_report(args.report) rows = extract_rows(report, args.tolerance) if args.write_baseline: write_baseline(rows) return 0 summary = build_summary(rows, tolerance) print(summary, flush=True) if args.summary_out: args.summary_out.parent.mkdir(parents=True, exist_ok=True) args.summary_out.write_text(summary, encoding="utf-8") print(f"Summary written to {args.summary_out}", flush=True) failed = [row for row in rows if row["status"] != STATUS_PASS] if not rows: print("No eval cases ran. Treating that as a failure.", file=sys.stderr) return 1 if failed: print( f"{len(failed)} of {len(rows)} eval cases failed the gate: " f"{', '.join(row['case_id'] for row in failed)}", file=sys.stderr, ) return 1 print(f"All {len(rows)} eval cases are within tolerance.") return 0if __name__ == "__main__": sys.exit(main())
Where Autonoma Fits in an LLM Eval Pipeline
LLM evals answer whether a response clears a quality bar. Autonoma covers the next gate: whether that response caused the correct result in the running product. Its Planner reads the code paths a feature touches and plans behavioral test cases for a live preview environment; its Executor drives the real UI and asserts on the resulting state, such as a record update, navigation, or side effect. Run that behavioral gate beside the eval jobs when a model response can trigger an action, and let Diffs Agent keep the affected coverage current as the application changes.
Threshold Gating: Averaging Runs So Noise Doesn't Fail Your Build
A single eval run is not a reliable signal, and treating it like one is the fastest way to make a team stop trusting the suite. Call the same model with the same prompt five times and the score will typically wobble, not because anything regressed, but because sampling temperature, token-level randomness, and small phrasing differences move a similarity or judge score by a few points in either direction on their own. Fail the build on any single low run, and you'll be re-running CI by hand within a week. Keep that up, and the team starts merging past red checks on principle.
The fix DeepEval's own docs get right, and that most teams skip anyway: run each case N times, average the score, and compare the average against a baseline minus a tolerance band, never any individual run against an exact target. Three runs is a floor. Five is a comfortable default for anything that isn't prohibitively expensive to call repeatedly. A tolerance of 0.05 to 0.1 on a 0-to-1 scale absorbs normal sampling noise without absorbing a real regression, since a genuine drop tends to drag every one of the N runs down together, not just one of them.
Exact match versus semantic similarity is the other half of this. Scoring "the refund was processed" against "I've processed your refund" with a string comparison manufactures a failure out of a correct answer. Scoring the same pair with an embedding-similarity or LLM-as-judge comparison against a reference answer correctly calls it a pass. If a case turns out flaky under this setup, meaning it passes on some runs and fails on others even at N=5, the honest read is one of two things: either the assertion is too strict for the range of correct phrasings the model can produce, or the prompt itself is ambiguous enough that the model's answer legitimately varies in substance, not just in wording. Both are real bugs. Neither one is in your code.
Same five-run average, same threshold. One scenario is sampling noise. The other is a real drop.
Here's the scorer that folds a semantic-similarity check and a groundedness check into one float, so the threshold comparison downstream has a single number to work with:
"""A custom eval scorer: semantic similarity plus optional groundedness.Two signals, deliberately kept separate:1. **Semantic similarity** — cosine distance between the embedding of the model's response and the embedding of a golden reference answer. Catches "the bot stopped saying the right thing".2. **Groundedness** — an LLM judge that reads the response against the supplied context and reports what fraction of the response's factual claims the context actually supports. Catches "the bot said something plausible that nobody wrote down", which similarity alone scores generously.`score_response` returns a single float in `[0.0, 1.0]` so the rest of thepipeline (tests, baselines, CI gate) only ever deals with one number.Nothing here is framework-specific. Swap `_embed` and `groundedness` for yourown provider and every caller keeps working."""from __future__ import annotationsimport jsonimport osfrom functools import lru_cachefrom typing import List, Optional, Sequenceimport numpy as npEMBEDDING_MODEL = os.environ.get("EVAL_EMBEDDING_MODEL", "text-embedding-3-small")GROUNDEDNESS_MODEL = os.environ.get("EVAL_GROUNDEDNESS_MODEL", "gpt-4o-mini")# How much of the combined score comes from similarity when a context is given.# The remainder comes from groundedness.SIMILARITY_WEIGHT = float(os.environ.get("EVAL_SIMILARITY_WEIGHT", "0.6"))GROUNDEDNESS_SYSTEM_PROMPT = ( "You are a strict grader measuring groundedness. You are given a CONTEXT " "and a RESPONSE. Break the RESPONSE into its distinct factual claims, then " "decide how many of those claims are directly supported by the CONTEXT. " "A claim is supported only if the CONTEXT states it or clearly implies it. " "Generic pleasantries, offers to help, and hedges are not factual claims " "and must be ignored. Reply with JSON only, no prose, in exactly this " 'shape: {"total_claims": <int>, "supported_claims": <int>, ' '"unsupported_claims": [<string>, ...]}')_client = Nonedef _get_client(): """Lazily build the OpenAI client so importing this module needs no key.""" global _client if _client is None: from openai import OpenAI # imported lazily: keeps import-time cost at zero api_key = os.environ.get("OPENAI_API_KEY") if not api_key: raise RuntimeError( "OPENAI_API_KEY is not set. Export it before scoring: " "export OPENAI_API_KEY=sk-..." ) _client = OpenAI(api_key=api_key) return _client@lru_cache(maxsize=2048)def _embed(text: str) -> tuple: """Embed one string. Cached, because reference texts repeat across N runs.""" client = _get_client() response = client.embeddings.create(model=EMBEDDING_MODEL, input=text) return tuple(response.data[0].embedding)def cosine_similarity(vec_a: Sequence[float], vec_b: Sequence[float]) -> float: """Cosine similarity clamped into `[0.0, 1.0]`. Real embedding pairs land in roughly `[0.0, 1.0]` already; the clamp exists so a rare negative never leaks a nonsensical score into a CI threshold. """ a = np.asarray(vec_a, dtype=np.float64) b = np.asarray(vec_b, dtype=np.float64) norm_a = float(np.linalg.norm(a)) norm_b = float(np.linalg.norm(b)) if norm_a == 0.0 or norm_b == 0.0: return 0.0 similarity = float(np.dot(a, b) / (norm_a * norm_b)) return max(0.0, min(1.0, similarity))def semantic_similarity(response: str, reference: str) -> float: """How close the response is in meaning to the golden reference answer.""" if not reference or not reference.strip(): raise ValueError("reference must be a non-empty string") if not response or not response.strip(): return 0.0 return cosine_similarity(_embed(response.strip()), _embed(reference.strip()))def _judge(response: str, context: str) -> dict: """Ask the judge model to grade `response` against `context`. Returns the parsed verdict: `total_claims`, `supported_claims`, and `unsupported_claims`. Temperature is pinned to 0 because a grader that disagrees with itself run to run adds noise to the very number you are trying to measure. """ client = _get_client() completion = client.chat.completions.create( model=GROUNDEDNESS_MODEL, temperature=0.0, response_format={"type": "json_object"}, messages=[ {"role": "system", "content": GROUNDEDNESS_SYSTEM_PROMPT}, { "role": "user", "content": f"CONTEXT:\n{context.strip()}\n\nRESPONSE:\n{response.strip()}", }, ], ) raw = completion.choices[0].message.content or "" try: return json.loads(raw) except json.JSONDecodeError as exc: raise RuntimeError( f"groundedness judge did not return JSON (model={GROUNDEDNESS_MODEL}): {raw!r}" ) from excdef groundedness(response: str, context: str) -> float: """Fraction of the response's factual claims that the context supports. Returns 1.0 when the response makes no factual claims at all (a pure hand-off or refusal invents nothing, so it cannot be ungrounded). """ if not context or not context.strip(): raise ValueError("context must be a non-empty string") if not response or not response.strip(): return 0.0 verdict = _judge(response, context) try: total = int(verdict["total_claims"]) supported = int(verdict["supported_claims"]) except (KeyError, TypeError, ValueError) as exc: raise RuntimeError( f"groundedness judge returned malformed JSON: {verdict!r}" ) from exc if total <= 0: # No factual claims at all. A pure hand-off or refusal invents nothing, # so there is nothing for the context to fail to support. return 1.0 return max(0.0, min(1.0, supported / total))def unsupported_claims(response: str, context: str) -> List[str]: """The judge's list of claims it could not find in the context. Useful when debugging a regression locally: the score tells you something broke, this tells you which sentence did it. """ if not context or not context.strip(): raise ValueError("context must be a non-empty string") if not response or not response.strip(): return [] claims = _judge(response, context).get("unsupported_claims") or [] return [str(claim) for claim in claims]def score_response( response: str, reference: str, context: Optional[str] = None,) -> float: """Score one model response against a reference. Returns 0.0 to 1.0. Without `context` the score is pure semantic similarity. With `context` the score is a weighted blend of similarity and groundedness, so a fluent answer that quietly invents a refund window scores below one that does not. """ similarity = semantic_similarity(response, reference) if context is None: return round(similarity, 4) grounded = groundedness(response, context) combined = SIMILARITY_WEIGHT * similarity + (1.0 - SIMILARITY_WEIGHT) * grounded return round(max(0.0, min(1.0, combined)), 4)
And here's the pytest suite that calls it: one test per dataset case, five invocations per test, one averaged assertion against the stored baseline and tolerance:
"""Model-backed eval tests. Slow, costs money, runs on merge-to-main and nightly.One parametrized test per case in `data/eval_dataset.jsonl`. Each test calls theapp N times (default 5), scores every run, and asserts on the **average**. Thataveraging is the whole point: a single unlucky sample from a non-deterministicmodel must not be able to turn CI red on its own. Only a real shift in thedistribution moves the average past the threshold.The threshold is per-case: `baseline_scores.json[case_id] - tolerance`. Absolutethresholds punish hard cases and let easy ones rot; a per-case baseline comparesthe model against its own last accepted behaviour.Each test attaches its scores to the `json_metadata` fixture so that`scripts/run_evals.py` can rebuild the full picture from the pytest JSON reportwithout re-running a single model call."""from __future__ import annotationsimport jsonimport osimport statisticsfrom pathlib import Pathfrom typing import Any, Dict, Listimport pytestfrom src.app import generate_responsefrom src.scorer import score_responseDATA_DIR = Path(__file__).resolve().parents[1] / "data"DATASET_PATH = DATA_DIR / "eval_dataset.jsonl"BASELINE_PATH = DATA_DIR / "baseline_scores.json"REQUIRED_CASE_KEYS = ("id", "input", "reference", "tags")# A case tagged "grounded" is scored with groundedness folded in, using its# reference text as the grounding context. Untagged cases are pure similarity.GROUNDED_TAG = "grounded"def load_dataset(path: Path = DATASET_PATH) -> List[Dict[str, Any]]: """Read the JSONL eval dataset, skipping blank lines.""" if not path.exists(): raise FileNotFoundError(f"eval dataset not found at {path}") cases: List[Dict[str, Any]] = [] with path.open("r", encoding="utf-8") as handle: for line_number, raw_line in enumerate(handle, start=1): line = raw_line.strip() if not line: continue try: case = json.loads(line) except json.JSONDecodeError as exc: raise ValueError(f"{path.name} line {line_number} is not valid JSON: {exc}") from exc missing = [key for key in REQUIRED_CASE_KEYS if key not in case] if missing: raise ValueError( f"{path.name} line {line_number} is missing keys: {', '.join(missing)}" ) cases.append(case) if not cases: raise ValueError(f"{path.name} contains no eval cases") return casesdef load_baseline(path: Path = BASELINE_PATH) -> Dict[str, Any]: """Read the recorded baseline scores.""" if not path.exists(): raise FileNotFoundError(f"baseline scores not found at {path}") with path.open("r", encoding="utf-8") as handle: baseline = json.load(handle) if "cases" not in baseline: raise ValueError(f"{path.name} must contain a 'cases' object") return baselineCASES = load_dataset()BASELINE = load_baseline()RUNS_PER_CASE = int(os.environ.get("EVAL_RUNS_PER_CASE", BASELINE.get("runs_per_case", 5)))TOLERANCE = float(os.environ.get("EVAL_TOLERANCE", BASELINE.get("tolerance", 0.05)))pytestmark = pytest.mark.skipif( not os.environ.get("OPENAI_API_KEY"), reason="OPENAI_API_KEY is not set, so model-backed evals cannot run",)@pytest.mark.parametrize("case", CASES, ids=[case["id"] for case in CASES])def test_eval_case(case: Dict[str, Any], json_metadata) -> None: case_id = case["id"] if case_id not in BASELINE["cases"]: pytest.fail( f"case '{case_id}' has no baseline. Add it to data/baseline_scores.json " f"or run: python scripts/run_evals.py --write-baseline" ) baseline = float(BASELINE["cases"][case_id]) threshold = baseline - TOLERANCE context = case["reference"] if GROUNDED_TAG in case["tags"] else None scores: List[float] = [] responses: List[str] = [] for _ in range(RUNS_PER_CASE): response = generate_response(case["input"], context=context) responses.append(response) scores.append(score_response(response, case["reference"], context=context)) average = round(statistics.fmean(scores), 4) # Read back out of the JSON report by scripts/run_evals.py. json_metadata["eval"] = { "case_id": case_id, "tags": case["tags"], "runs": RUNS_PER_CASE, "scores": scores, "average": average, "min": min(scores), "max": max(scores), "baseline": baseline, "tolerance": TOLERANCE, "threshold": round(threshold, 4), "grounded": context is not None, "worst_response": responses[scores.index(min(scores))], } assert average >= threshold, ( f"'{case_id}' regressed: average {average:.4f} over {RUNS_PER_CASE} runs " f"is below the threshold {threshold:.4f} (baseline {baseline:.4f} " f"minus tolerance {TOLERANCE:.4f}).\n" f" individual scores: {scores}\n" f" worst response: {responses[scores.index(min(scores))]!r}" )
Notice what this test does not do. It never asserts that a single run matches a target exactly, and it never fails on one below-average outlier on its own. Non-determinism gets absorbed by the average. A real regression, consistent across all five runs, still gets caught, because there's nowhere for it to hide once five numbers are being averaged instead of one noisy one being trusted on its own.
Storing and Versioning the Eval Dataset
An eval suite is only as good as the dataset it runs against, and the most common way teams sabotage their own pipeline is pasting golden answers straight into the test file, where nobody outside the person who wrote them can review a change to one. Treat the dataset the way you'd treat a database migration: a versioned file in the repo, reviewed in a pull request, with a schema everyone on the team actually understands.
A workable schema is small on purpose: an id, the input, a reference answer or grounding context, and the tags a case needs, which query type it represents, which past regression it was added to catch. Store it as JSON Lines, one case per line, so adding a new case is a one-line diff and a reviewer can see exactly what changed without a data-tooling detour.
{"id": "refund-window-standard", "input": "How long do I have to return something I bought?", "reference": "Northwind Supply accepts returns within 30 days of delivery for a full refund, as long as the item is unused and in its original packaging. Refunds are issued to the original payment method and take 5 to 7 business days to appear.", "tags": ["policy", "returns", "grounded", "regression-refund-window-shortened-to-14"]}{"id": "refund-window-opened-item", "input": "I opened the box and used the drill once. Can I still return it?", "reference": "Opened or used items are not eligible for a full refund. Northwind Supply offers store credit for opened items returned within 30 days of delivery, minus a 15% restocking fee. Items showing damage beyond normal inspection are not accepted.", "tags": ["policy", "returns", "edge-case", "grounded"]}{"id": "shipping-time-domestic", "input": "When will my order get here if I'm in Ohio?", "reference": "Standard domestic shipping arrives in 3 to 5 business days. Expedited shipping arrives in 2 business days and costs $12.99. Orders placed after 2pm ET ship the next business day.", "tags": ["shipping", "factual-lookup", "grounded"]}{"id": "shipping-international-fees", "input": "Do you ship to Canada, and who pays customs?", "reference": "Northwind Supply ships to Canada with delivery in 7 to 12 business days. International shipping costs $24.99 flat. The customer is responsible for any customs duties, import taxes, and brokerage fees charged on arrival.", "tags": ["shipping", "international", "grounded", "regression-invented-free-customs"]}{"id": "billing-double-charge", "input": "I got charged twice for order 88213. What happens now?", "reference": "Duplicate charges are usually an authorization hold rather than a second capture, and holds drop off within 3 to 5 business days. If both charges have settled, Northwind Supply reverses the duplicate within 1 business day of the report and the bank posts it in 5 to 7 business days. The customer does not need to return anything.", "tags": ["billing", "multi-step", "grounded"]}{"id": "billing-subscription-cancel", "input": "How do I cancel my Northwind Plus membership and do I get money back?", "reference": "Northwind Plus can be cancelled at any time from Account Settings under Memberships. Cancellation takes effect at the end of the current billing period, and the membership stays active until then. Annual memberships cancelled within 14 days of purchase are refunded in full.", "tags": ["billing", "account", "grounded"]}{"id": "account-password-reset", "input": "I can't log in and the reset email never shows up.", "reference": "Password reset links are sent immediately and expire after 60 minutes. If the email has not arrived, check the spam folder and confirm the address on file matches the one being used. Accounts are locked for 30 minutes after five failed login attempts, and the lock clears on its own.", "tags": ["account", "troubleshooting", "grounded"]}{"id": "order-status-unknown-number", "input": "What's the status of order 99999999?", "reference": "That order number is not in the system. The assistant should say it cannot find the order rather than guessing, ask the customer to confirm the number from their confirmation email, and offer to hand the conversation to a human agent.", "tags": ["negative-case", "hallucination-guard", "regression-invented-tracking-number"]}{"id": "out-of-scope-investment", "input": "My order was late. What should I invest the refund in?", "reference": "I can only help with questions about your account, orders, billing, and shipping. For anything else, please contact a human agent at support@example.com.", "tags": ["out-of-scope", "refusal", "regression-refusal-string-drift"]}
Golden-answer drift is the maintenance cost nobody budgets for. A reference answer written against last quarter's product copy, last quarter's pricing, or last month's policy becomes wrong in a way that has nothing to do with the model, and a stale reference will flag a perfectly correct answer as a regression indefinitely, until someone happens to catch it. Give the dataset an owner, review it on the same cadence you'd review any other test fixture, and treat "is this reference still true" as its own checklist item whenever the product copy it's testing against changes.
New cases should come from two sources, not one. Some get written up front, covering the query types the product is known to handle. The rest should come from production: a support escalation, a confusing transcript, a case where the model gave a confident but wrong answer. Turn each of those into a dataset row with the correct reference answer attached, and the eval suite grows in the direction your actual users are pushing it, instead of only covering what someone imagined at design time. A dataset that only ever grows from the design phase stops testing the failure modes that show up once real traffic hits the feature.
None of this replaces the harder problem one layer up: nothing here tells you whether an agent kept the same tool-call decision it made last week, only that a fixed set of eval cases still scores where it should. If what you actually need to catch is drift after a silent model update rather than a general quality bar, agent regression testing is the pattern built specifically for that, golden trajectories instead of golden answers, and the same N-run averaging idea applied to a tool-call sequence instead of a text score.
The prompt-level unit tests this pipeline gates on every commit, the deeper non-determinism patterns underneath the averaging above, and the guardrail and prompt-injection tests worth running alongside your evals are each their own piece; this one is about the CI wiring that ties all of them into a single pipeline. Add Autonoma when that pipeline also needs to verify the user-visible action and application state a passing model response produced.
Frequently Asked Questions
Split the pipeline into two lanes. Deterministic tests (prompt building, response parsing, retries, schema checks) run on every commit and pull request because they are free and fast. The LLM eval suite runs on merge to main plus a nightly cron, since each case costs paid model tokens and seconds. Each eval case runs N times (five is a common default), the scores are averaged, and the average is compared against a stored baseline within a tolerance band rather than failing on any single low run.
LLM evals are automated tests that score a model's output against a quality bar, such as correctness, faithfulness to a source document, or tone, using a scoring function like semantic similarity or LLM-as-judge instead of an exact-match assertion. They differ from unit tests because the thing being checked is a graded quality, not a fixed, reproducible output.
A minimal eval framework needs four pieces: a versioned dataset of input/reference-answer pairs stored as reviewable files, a scoring function that grades a fresh response against the reference (semantic similarity, faithfulness, or an LLM-as-judge call), a runner that executes each case multiple times and averages the score, and a CI job that gates a threshold check on merges and a nightly schedule rather than every commit.
You don't force the output itself to be reproducible. You run each test case multiple times (five is a common default), average the score across those runs, and compare the average against a baseline within a tolerance band. A single run is not a reliable signal; an averaged score absorbs normal sampling noise while still catching a genuine, consistent regression.
Replace exact string matching with a scoring function: embedding-based semantic similarity against a reference answer, a faithfulness or groundedness check against retrieved context, or an LLM-as-judge call that rates the response against a rubric. The score is compared to a threshold rather than to an exact string, since correct answers can be worded many different ways.
No. Eval tests call a paid model and take seconds per case, so running the full suite on every commit is slow and expensive. Run deterministic tests (parsing, retries, schema checks) on every commit, and run the LLM eval suite on merge to main plus a nightly cron, which catches both deliberate prompt or model changes and silent provider-side model updates.
