Agent Regression Testing: When Your Agent Breaks Without a Deploy

Tuesday's dashboard read 93% routing accuracy, the number our support agent had held steady for six weeks. Thursday it read 71%. Nobody had opened a pull request. Nobody had merged anything. The deploy log for that window was empty.

The model provider had swapped the LLM behind the API endpoint we called, quietly, mid-week, with no changelog entry any QA process would think to watch. Same prompt. Same tools. Same "test suite," if you could call it that, a handful of exact-string assertions against old transcripts, and every one of those assertions still passed, because the agent still replied in English and still called a tool. It just called the wrong one, more often, in ways a string match will never see.

That's the failure mode this piece is about: a regression with nothing to do with your code.

The Silent Model Update That Cost 22 Points of Accuracy

The setup was mundane, which is exactly why it's worth using as the example. A support-routing agent read an incoming ticket and decided which of four downstream queues it belonged in, billing, technical, account access, or escalate-to-human, then called the matching tool. We measured it the only way that matters: run a labeled set of past tickets through the agent, check whether the tool call it picked matches the human-labeled correct queue, average the hits. That number sat at 93% for six weeks, comfortably above the 90% floor the support team had agreed was acceptable.

Then it dropped to 71%, and the drop showed up as a wave of mis-routed billing tickets landing in the technical queue, not in any dashboard we were watching. When we went looking for what shipped, the answer was nothing. Our prompt hadn't changed. Our tool definitions hadn't changed. Our knowledge base hadn't changed. The one thing that had changed, invisibly, was the model sitting behind the API endpoint the agent called. The provider rolled a new version into the same model name, the routing behavior it had learned to imitate shifted underneath it, and every regression gate we had, all of them keyed to code changes, had nothing to trigger on.

Why Traditional Regression Testing Doesn't Catch This

Traditional regression testing rests on three assumptions that quietly stop holding once an LLM sits inside the loop. It assumes a change worth testing is a change your team made, so it wires itself to a merge or a deploy hook. It assumes the same input produces the same output, so a snapshot diff or an exact string match is a valid pass or fail signal. And it assumes "correct" is a property of the text itself, checkable by comparing it to a stored answer.

None of those hold for an agent built on a model whose weights you don't control. The provider can update the model behind an API name with no changelog entry your CI would ever parse, which means llm regression testing has to run on a schedule, not a hook, because there is no commit to trigger it. The same prompt against the same model produces different phrasing on every call, so an exact-match assertion will fail constantly on entirely correct output, training every engineer on the team to ignore the test within a month. And correctness for an agent usually lives in the decision, which tool it called, which record it touched, not in the words wrapped around that decision, so grading the text alone misses exactly the kind of regression that took our routing accuracy from 93 to 71.

The 2026 arXiv paper on AgentAssay (arXiv 2603.02601) frames this precisely: token-efficient regression testing for non-deterministic agent workflows means comparing structured execution traces, not raw text, and doing it cheaply enough to run continuously instead of as a one-off audit. That's the right instinct. The rest of this piece is a runnable, vendor-neutral version of it.

Golden Trajectories: Recording What "Good" Looked Like

A golden trajectory is a recording of one input run through the agent at a moment when a human checked the result and confirmed it was correct. Not just the final reply, the whole path: which tools got called, in what order, with what arguments, and what the agent said at the end. Capture the input, the tool-call sequence, and the final output together, and you have a baseline artifact every future run gets compared against, model update or not.

The discipline that matters here is capturing the trajectory, not just the outcome. "The ticket got routed to billing" is one bit of information. "The agent read the ticket, called classify_intent, got back billing at 0.94 confidence, then called route_to_queue("billing")" is the actual shape of the decision, and it's the shape that breaks first when a model update shifts behavior, often before the final label changes at all.

Here's a capture script that wraps a single agent invocation and writes exactly that shape to a baseline file, using a plain callable so it works against any agent framework underneath:

"""Capture a golden agent trajectory for regression testing.

An agent has no application "deploy" event to hang a regression gate on: the
model can change under you, a prompt edit ships instantly, a knowledge-base
file changes retrieval. So instead of gating on a build, we gate on behavior.
This script records a *golden trajectory* the moment a human has confirmed the
agent did the right thing, and writes it to ``golden_trajectories/<case_id>.json``.
Commit that file next to the test suite; the tests replay each case and compare.

Usage::

    python src/capture_baseline.py support-routing-billing

Run it once per labeled case to build the ``golden_trajectories/`` directory.

This file is vendor-neutral: it depends on nothing but the standard library.
The agent itself is any callable you supply that takes an input string and
returns ``{"tool_calls": [...], "output_text": "..."}``. Swap ``demo_agent``
below for your real agent and extend ``CASES`` with your labeled inputs.
"""

from __future__ import annotations

import json
import sys
from pathlib import Path

# golden_trajectories/ lives at the repo root, one level above src/.
GOLDEN_DIR = Path(__file__).resolve().parent.parent / "golden_trajectories"


def invoke_agent(agent_fn, input_text):
    """Run a user-supplied agent callable and normalize its trajectory.

    ``agent_fn`` is any callable taking a single input string and returning a
    dict with:

    - ``tool_calls``: a list of ``{"name", "args", "result"}`` dicts, in the
      order the agent called them.
    - ``output_text``: the agent's final natural-language answer.

    Returns a normalized ``{"input", "tool_calls", "output_text"}`` dict that is
    safe to serialize as a golden trajectory. This is the single shim every
    other file in the repo goes through, so capture and testing stay identical.
    """
    result = agent_fn(input_text)
    if not isinstance(result, dict):
        raise TypeError(
            f"agent_fn must return a dict, got {type(result).__name__}"
        )
    if "tool_calls" not in result or "output_text" not in result:
        raise ValueError(
            "agent_fn result must contain 'tool_calls' and 'output_text'"
        )

    normalized_calls = []
    for call in result["tool_calls"]:
        normalized_calls.append(
            {
                "name": call["name"],
                "args": call.get("args", {}),
                "result": call.get("result"),
            }
        )

    return {
        "input": input_text,
        "tool_calls": normalized_calls,
        "output_text": result["output_text"],
    }


# ---------------------------------------------------------------------------
# DEMO AGENT + CASE REGISTRY
#
# ``demo_agent`` is a deterministic stand-in so this repo runs out of the box
# with zero dependencies and zero API keys. Replace it with your real agent
# (a LangChain/LlamaIndex/OpenAI/Anthropic call, an internal orchestrator,
# whatever) and add your own labeled cases to CASES.
# ---------------------------------------------------------------------------


def demo_agent(input_text):
    """A fake support-routing agent that classifies, looks up, and routes.

    It returns a fixed trajectory so the captured baseline is reproducible.
    A real agent would make live model + tool calls here.
    """
    return {
        "tool_calls": [
            {
                "name": "classify_intent",
                "args": {"text": input_text},
                "result": {"intent": "billing", "confidence": 0.94},
            },
            {
                "name": "lookup_customer",
                "args": {"email": "user@example.com"},
                "result": {"customer_id": "cus_10482", "plan": "pro"},
            },
            {
                "name": "route_to_queue",
                "args": {"queue": "billing", "priority": "normal"},
                "result": {"queue": "billing", "ticket_id": "T-55021"},
            },
        ],
        "output_text": (
            "I've routed your duplicate-charge issue to our billing team. "
            "Ticket T-55021 is open and you'll hear back within one business day."
        ),
    }


CASES = {
    "support-routing-billing": {
        "input": "I was charged twice for my subscription this month, can you help?",
        "agent_fn": demo_agent,
    },
}


def capture(case_id):
    """Run one labeled case through its agent and write the golden trajectory."""
    if case_id not in CASES:
        known = ", ".join(sorted(CASES)) or "(none)"
        raise KeyError(f"Unknown case '{case_id}'. Known cases: {known}")

    case = CASES[case_id]
    trajectory = invoke_agent(case["agent_fn"], case["input"])

    GOLDEN_DIR.mkdir(parents=True, exist_ok=True)
    out_path = GOLDEN_DIR / f"{case_id}.json"
    out_path.write_text(json.dumps(trajectory, indent=2) + "\n", encoding="utf-8")
    return out_path


def main(argv):
    if len(argv) != 2:
        print("usage: python src/capture_baseline.py <case_id>", file=sys.stderr)
        print(f"known cases: {', '.join(sorted(CASES)) or '(none)'}", file=sys.stderr)
        return 2

    case_id = argv[1]
    try:
        out_path = capture(case_id)
    except (KeyError, ValueError, TypeError) as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 1

    print(f"wrote golden trajectory: {out_path}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Run this once against every case in a labeled set, while the agent is behaving the way you want it to, and check the resulting JSON files into the repo next to the test suite. That directory, not a fresh model call, is what everything downstream compares against.

Scoring a Fresh Run Against the Golden Baseline

Comparing a fresh run to a golden trajectory needs two different measurements, because the two halves of a trajectory fail in different ways. The tool-call sequence is a hard fact: either the agent called classify_intent then route_to_queue("billing"), or it didn't, and that half of the comparison should be an exact structural match, not a fuzzy one. The final text reply is not a hard fact. "Routed to billing" and "I've sent this over to the billing team" are both correct, and scoring them with an exact match manufactures regressions out of nothing. That half needs a similarity score, an embedding comparison, a string-similarity ratio, or an LLM-as-judge call, folded into one number alongside the structural half.

Here's a scorer that combines both into a single float between 0 and 1, so a threshold check downstream has one value to compare against the baseline:

"""Score a fresh agent trajectory against a golden baseline.

The score has two halves, because an agent can go wrong in two different ways:

1. **Structural (weight 0.6)** - did it call the right tools, with the right
   arguments, in the right order? This is a hard, binary signal: any deviation
   in the ordered sequence of ``(name, args)`` pairs scores 0. We deliberately
   compare *names and arguments*, not tool *results*, because results often
   carry non-deterministic values (fresh ticket IDs, timestamps) that say
   nothing about whether the agent reasoned correctly.

2. **Text (weight 0.4)** - how close is the final answer to the golden answer?
   By default this uses :class:`difflib.SequenceMatcher`, which is pure standard
   library: zero dependencies and zero API keys to run in CI. In production you
   almost certainly want a semantic comparison instead, so ``similarity_fn`` is
   pluggable: pass any ``(str, str) -> float in [0, 1]`` callable, e.g. an
   embedding cosine similarity or an LLM-as-judge call.

``python src/scorer.py`` runs a standalone demo comparing the shipped golden
file to a reworded fake fresh run and prints the resulting score.
"""

from __future__ import annotations

import difflib
import json
import sys
from pathlib import Path

STRUCTURAL_WEIGHT = 0.6
TEXT_WEIGHT = 0.4

GOLDEN_DIR = Path(__file__).resolve().parent.parent / "golden_trajectories"


def _default_similarity(a: str, b: str) -> float:
    """Stdlib text similarity in [0, 1]. No deps, no network, no keys."""
    return difflib.SequenceMatcher(None, a, b).ratio()


def _tool_signature(call):
    """Reduce a tool call to the part the agent actually decided.

    Name + arguments (order-sensitive across the list), ignoring the tool's
    returned ``result`` which may be non-deterministic.
    """
    return (call.get("name"), json.dumps(call.get("args", {}), sort_keys=True))


def score_trajectory(golden, fresh, similarity_fn=None) -> float:
    """Return a weighted 0-1 score comparing ``fresh`` against ``golden``.

    ``golden`` and ``fresh`` are trajectory dicts with ``tool_calls`` and
    ``output_text`` (as produced by ``invoke_agent`` in capture_baseline.py).
    Perfect match returns 1.0.
    """
    if similarity_fn is None:
        similarity_fn = _default_similarity

    golden_sig = [_tool_signature(c) for c in golden.get("tool_calls", [])]
    fresh_sig = [_tool_signature(c) for c in fresh.get("tool_calls", [])]
    structural = 1.0 if golden_sig == fresh_sig else 0.0

    text = similarity_fn(golden.get("output_text", ""), fresh.get("output_text", ""))
    text = max(0.0, min(1.0, float(text)))

    return STRUCTURAL_WEIGHT * structural + TEXT_WEIGHT * text


def _demo():
    """Compare the shipped golden file to a reworded fake fresh trajectory."""
    golden_path = GOLDEN_DIR / "support-routing-billing.json"
    golden = json.loads(golden_path.read_text(encoding="utf-8"))

    # Same tool calls (structural match), but a reworded final answer so the
    # text half lands below 1.0 and you can see how the halves combine.
    fresh = {
        "input": golden["input"],
        "tool_calls": golden["tool_calls"],
        "output_text": (
            "Your double-charge has been sent to the billing team. "
            "Ticket T-55021 was created; expect a reply within a business day."
        ),
    }

    score = score_trajectory(golden, fresh)
    print(f"golden vs golden : {score_trajectory(golden, golden):.3f}")
    print(f"golden vs fresh  : {score:.3f}")
    print(
        "structural halves match (0.600); the text half is discounted "
        "because the wording differs."
    )
    return 0


if __name__ == "__main__":
    raise SystemExit(_demo())

That single score is what feeds the drift check. It's also, and this matters for what comes later, a score about the conversation, not about the application the agent's tool calls are supposed to affect. A perfect score here confirms the agent said the right thing and, per the structural half, called the right tool. It does not confirm that route_to_queue("billing") actually moved the ticket in the system the support team looks at, or that the technical queue's dashboard reflects the change. Those are two different verification problems, and it's entirely possible to grade the first one perfectly while the second one silently breaks underneath it, which is exactly the gap Autonoma exists to close, a point worth coming back to once the regression suite itself is built.

Averaging N Runs So Noise Isn't Mistaken for Regression

Every agent invocation in this harness is an LLM call, which means a single fresh run can score lower than the golden baseline for reasons that have nothing to do with a real regression. The model phrased something slightly differently. The similarity function scored a correct paraphrase a little harshly. None of that is the 93-to-71 drop from earlier. It's noise, and treating one low-scoring run as a failed test teaches the team to ignore the test within a week, the same trap as the exact-match assertion from two sections ago.

The fix is to run each golden case N times, average the score across those N runs, and compare the average, not any individual run, against the baseline minus a tolerance threshold. Five runs and a 0.05 tolerance band is a reasonable starting point: run the case five times, average the five scores, and only fail if that average drops more than 0.05 below what the baseline scored. A single bad run out of five barely moves the average. A genuine regression, the kind that would have caught our routing agent's provider swap, drags every one of the five runs down together, and the average reflects that unambiguously.

Here's the pytest suite that does exactly this, one test per golden case, N invocations per test, one averaged assertion:

"""Averaged regression test for agent trajectories.

Agents are non-deterministic: the same input can yield slightly different
wording, and occasionally a different tool path, run to run. If you gate on a
single run you get a flaky, boy-who-cried-wolf suite. So this test runs each
case N times (default 5, override with the ``AGENT_REGRESSION_RUNS`` env var),
scores every run against the committed baseline, and asserts on the *average*.
One unlucky low run cannot fail the build; only a sustained drop does.

Run it with::

    pytest tests/test_regression.py -v
    AGENT_REGRESSION_RUNS=10 pytest tests/test_regression.py -v

Every ``golden_trajectories/*.json`` file becomes its own parametrized test.
"""

from __future__ import annotations

import json
import os
from pathlib import Path

import pytest

from capture_baseline import invoke_agent
from scorer import score_trajectory

GOLDEN_DIR = Path(__file__).resolve().parent.parent / "golden_trajectories"

# Number of times to replay each case before averaging. Bump it in CI (via the
# env var) if your agent is especially noisy.
RUNS = int(os.environ.get("AGENT_REGRESSION_RUNS", "5"))

# How far the averaged score may fall below the baseline before we call it a
# regression. 0.05 tolerates normal wording jitter but catches real drift.
TOLERANCE = 0.05


# ---------------------------------------------------------------------------
# STUB AGENT - REPLACE THIS WITH YOUR REAL AGENT.
#
# This stub replays the golden trajectory so the suite is green out of the box.
# Point it at your live agent (model call + tools) and the exact same averaging
# machinery will catch prompt, model, or knowledge-base drift.
# ---------------------------------------------------------------------------
def agent_fn(input_text):
    case_id = _input_to_case_id.get(input_text)
    golden = json.loads((GOLDEN_DIR / f"{case_id}.json").read_text(encoding="utf-8"))
    return {
        "tool_calls": golden["tool_calls"],
        "output_text": golden["output_text"],
    }


def _golden_files():
    return sorted(GOLDEN_DIR.glob("*.json"))


# Map each golden input back to its case id so the stub can find its baseline.
# A real agent_fn ignores this entirely and just answers the input.
_input_to_case_id = {
    json.loads(p.read_text(encoding="utf-8"))["input"]: p.stem
    for p in _golden_files()
}


@pytest.mark.parametrize("golden_path", _golden_files(), ids=lambda p: p.stem)
def test_trajectory_does_not_regress(golden_path):
    golden = json.loads(golden_path.read_text(encoding="utf-8"))

    # The ceiling: the golden trajectory scored against itself (1.0 for the
    # default scorer). Deriving it keeps the assertion honest if the scorer's
    # weighting ever changes.
    baseline_score = score_trajectory(golden, golden)

    scores = []
    for _ in range(RUNS):
        fresh = invoke_agent(agent_fn, golden["input"])
        scores.append(score_trajectory(golden, fresh))

    average = sum(scores) / len(scores)

    assert average >= baseline_score - TOLERANCE, (
        f"{golden_path.stem}: averaged score {average:.3f} over {RUNS} runs "
        f"dropped more than {TOLERANCE} below baseline {baseline_score:.3f}. "
        f"per-run scores: {[round(s, 3) for s in scores]}"
    )

Notice what this test does not do: it does not assert that any single run matches the baseline exactly, and it does not treat a below-baseline outlier as a failure on its own. Non-determinism gets absorbed by the average. A real regression, model-wide and consistent across every run, still gets caught, because there's nowhere for it to hide once you're comparing five averaged numbers instead of one noisy one.

How Autonoma Tests What the Model Change Actually Did

Everything above answers one question well: did the agent's decision and its wording hold up against a known-good baseline. It answers a different question not at all: did the thing the agent's tool call was supposed to do actually happen in the application your users and your support team look at. A perfect regression score on route_to_queue("billing") confirms the agent chose to call that function. It does not confirm the ticket's status actually flipped in the admin dashboard, that the technical team's queue count actually went down, or that a stale row didn't get left behind in a table the tool call never touched. That gap is invisible from inside a trajectory, because a trajectory only ever records what the agent claims it did.

That's the layer we built Autonoma to cover, and it runs on a different verification path entirely, not a bigger scorer. The Planner reads your codebase, the routes, the queue tables, the admin screens a support lead actually opens, and plans behavioral end-to-end test cases against what the application really does, including generating the database state each scenario needs before it runs. The Execution Agent drives the UI in an isolated PreviewKit environment, Autonoma's managed preview-environments layer, clicking through the actual routing flow instead of reading a transcript about it. The GenerationReviewer classifies what it finds, a real bug, an agent error, or a plan that no longer matches the current app. And the Diffs Agent keeps the whole suite aligned to the codebase on every pull request, adding, deprecating, and maintaining test cases as the code evolves, so these tests don't silently rot the next time someone reworks the queue-routing endpoint.

Put the two side by side on the exact scenario from earlier: the pytest suite from the sections above proves the agent still picks the right tool and phrases the outcome correctly after a model update, averaged across five runs to rule out noise. Autonoma's behavioral layer proves the ticket's status genuinely changed to "routed: billing" in the dashboard the support team is staring at, that the technical queue's count actually dropped, and that a regression introduced next month, someone reworks the queue-routing endpoint and nobody touches the agent's prompt, gets caught even though the trajectory-level score never moves. Neither one substitutes for the other. A response-level score and a behavioral check on the running app are answering two different questions, and the 93-to-71 drop this piece opened with is a response-level problem. The application-state problem is just as real, and it's the one a trajectory diff will never see.

Wiring the Threshold Gate Into CI

The pytest suite above is only useful if something actually runs it and someone actually finds out when it fails. That's a CI job, and the trigger for that job is the part that looks different from every other regression gate on the team, because there's no deploy or merge to hook it to. The model can change underneath you on a Tuesday afternoon with no pull request attached.

Here's the GitHub Actions workflow: it runs the regression suite, fails the job if the threshold assertion trips, and posts an alert on failure so drift surfaces the same day instead of sitting quietly in a log nobody opens:

name: agent-regression

# An AI agent has no application "deploy" to gate on. Behavior changes when the
# provider silently updates a model, when someone edits a prompt, or when a
# knowledge-base file changes. So we gate on two triggers instead:
#
#   1. A nightly cron, to catch silent provider-side model updates that ship
#      with no commit of ours at all.
#   2. Any push / PR that touches the things that actually change agent
#      behavior: model config, prompts, or knowledge-base files.
on:
  schedule:
    - cron: '0 6 * * *'
  push:
    paths:
      - 'config/model.yml'
      - 'prompts/**'
      - 'knowledge_base/**'
  pull_request:
    paths:
      - 'config/model.yml'
      - 'prompts/**'
      - 'knowledge_base/**'
  workflow_dispatch: {}

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install -r requirements.txt

      # The suite already fails on its own drift assertion (averaged score
      # falling below baseline - tolerance), so no extra threshold logic here.
      - name: Run agent regression suite
        run: pytest tests/test_regression.py -v

      # There is no deploy pipeline to surface a red build, so on failure we
      # push a same-day alert to a webhook. Store the URL as a repo secret named
      # REGRESSION_WEBHOOK_URL (Settings -> Secrets and variables -> Actions).
      - name: Alert on drift
        if: failure()
        env:
          REGRESSION_WEBHOOK_URL: ${{ secrets.REGRESSION_WEBHOOK_URL }}
        run: |
          if [ -z "${REGRESSION_WEBHOOK_URL}" ]; then
            echo "REGRESSION_WEBHOOK_URL secret is not set; skipping alert." >&2
            exit 0
          fi
          curl --fail --silent --show-error \
            -X POST "${REGRESSION_WEBHOOK_URL}" \
            -H 'Content-Type: application/json' \
            -d "{\"text\": \"Agent regression suite failed on ${GITHUB_REPOSITORY}@${GITHUB_SHA} (run ${GITHUB_RUN_ID}). Trajectory drift detected.\"}"

Scheduling: Nightly, and On Every Model, Prompt, or KB Change

Since there is no deploy event to hook a regression gate to, the schedule has to do the work a merge trigger normally would. Two triggers cover the failure modes that actually happen. The first is a nightly cron run, which exists purely to catch a silent provider-side model update, the exact failure that opened this piece, where nothing in your own repository changed at all. The second is a path-filtered trigger on anything your team does control: the model or provider config file, the prompt templates, the knowledge-base documents the agent retrieves from. Any of those is a real, deliberate change, and it deserves the same gate a code merge would get.

Skipping either trigger leaves a real gap. Schedule only the path-filtered run, and a silent model swap goes unnoticed until someone notices the support queue backing up, the way ours did. Schedule only the nightly run, and a genuinely bad prompt edit sits in the codebase for up to 24 hours before anyone finds out, instead of failing the same pull request that introduced it. Running both costs a few extra minutes of CI time a day. It's the cheapest insurance in this whole setup.

Building the Habit: Regression Testing as a Standing Practice

None of this is a one-time setup. A golden trajectory captured today reflects what "good" meant against today's model, today's prompt, and today's knowledge base. Add a new intent the agent needs to route, and it needs its own golden case. Change the prompt in a way that deliberately shifts behavior, and the old baselines need a human to re-check them before they get trusted again, or the suite will flag a deliberate improvement as a regression and nobody will believe the gate the next time it fires for real.

The teams that keep this running treat the golden set the way they'd treat a schema migration: reviewed in a pull request, not edited silently. When someone adds a case or updates a baseline, that's a diff a teammate looks at, the same scrutiny a change to a database migration would get, because a badly captured golden trajectory is worse than no regression suite at all. It teaches everyone to distrust the gate, and a gate nobody trusts is a gate nobody checks before merging.

The pattern in this piece, golden trajectories captured from known-good runs, a structural-plus-similarity scorer, N-run averaging to separate signal from noise, and a nightly-plus-on-change CI gate, is the foundation for testing an agent's decisions. It has a natural neighbor one layer up: how to test an AI agent covers the broader test pyramid an agent needs beyond regression alone, and AI agent reliability testing covers what happens once that agent is live and the questions shift from "did it regress" to "can I trust it in production." Autonoma adds the complementary application-state check: it verifies that a model decision produced the right visible result in the running product, on the isolated PR environment where the change will be reviewed. For running the eval layer this regression suite builds on top of inside your existing pipeline, how to run LLM evals in CI/CD is the next piece in this series.

Frequently Asked Questions