How to debug AI in production: trace one live request through its full call chain, retrieval, prompt assembly, model call, tool call, and response parsing, to find which step produced the wrong behavior. Then score a random sample of live traffic to confirm the pattern is real and not one anecdote. Unlike a normal bug there is rarely a stack trace: the request returned 200, the response read as plausible, and the failure only shows up in what the feature actually did once it reached a real user.
The feature had been live six weeks without incident before a user told support their refund had gone through. Finance had no record of any refund. Support pulled the request log for that conversation. Status 200. The response body was a clean, confident sentence confirming money was on its way. Nothing in the log said anything had gone wrong, because as far as the server was concerned, nothing had.
Why Production AI Failures Do Not Look Like Normal Bugs
A normal production bug announces itself. An exception bubbles up, a status code turns red, an alert fires because an error rate crossed a threshold. None of that happened here. The HTTP request completed. The model returned a response. The tool call the agent chose to make executed without raising anything. Every layer of the stack did exactly what it was told, and the result was still wrong.
That's the part that makes AI features different to debug. The failure isn't an exception, it's a decision: the wrong tool got called, the retrieved context didn't actually support the claim in the answer, the model filled a gap with something confident and false. Conventional AI application monitoring, a dashboard built to watch latency, error rate, and CPU, will show all-green through the entire incident, because none of those metrics measure whether the content or the action was correct. You find out the feature is broken from a support ticket, not a page.
How to Log AI Agent Behavior: Tracing the Call Chain
Every AI feature worth debugging is a chain, not a single function call: something retrieves context, something assembles a prompt from that context, something calls the model, the model's output may trigger a tool call, and something parses the result into what the user sees. A wrong answer can originate at any one of those points, and treating the whole chain as one opaque box is why "the AI is being weird" incidents drag on for hours. The fix is the same one distributed systems debugging arrived at years ago: instrument every boundary, not just the edges of the request. That is what observability for AI agents has to mean in practice: not a dashboard of averages, but stage-level evidence of what the feature retrieved, decided, and did, captured well enough to replay after the fact.
Six steps, in order. Jumping straight to rollback without localizing the step is how the same bug ships twice.
That overview is the shape the rest of this piece follows. The first two steps, reproduce and trace, live in the same place: structured logs at every stage boundary, timestamped, with enough of the input and output captured to tell which stage actually deviated from what a correct run looks like. Naming those stage attributes consistently is easier if you follow OpenTelemetry's GenAI semantic conventions rather than inventing your own field names.
Here's a pytest module that wraps a synthetic version of that five-stage chain, retrieval, prompt assembly, model call, tool call, response parsing, in structured log events and asserts each stage produced a well-formed result before the next stage runs:
"""Per stage trace assertions for an AI feature call chain.Run it: pytest tests/test_trace_assertions.py -vThe suite replays recorded production requests through the instrumented chain insrc/ai_pipeline.py. Each stage asserts its own output is well formed before thenext stage consumes it, so a failure names the stage that broke instead of onlytelling you the final answer was wrong.To watch the localized failure output for yourself, run: pytest tests/test_trace_assertions.py -v --runxfailThat turns the deliberately failing case at the bottom of this file into a realred test, and pytest prints the stage plus the payload it captured."""from __future__ import annotationsimport jsonfrom pathlib import Pathimport pytestfrom ai_pipeline import STAGES, StageAssertionError, run_chainFIXTURE = Path(__file__).resolve().parents[1] / "fixtures" / "recorded_chain_requests.json"def load_recorded(request_id: str) -> dict: with FIXTURE.open(encoding="utf-8") as handle: return json.load(handle)["requests"][request_id]def all_recorded() -> dict: with FIXTURE.open(encoding="utf-8") as handle: return json.load(handle)["requests"]def test_fixture_covers_every_stage_as_a_failure_site(): """Every stage should have at least one recorded request that breaks in it.""" failure_stages = {r["expected_failure_stage"] for r in all_recorded().values()} failure_stages.discard(None) assert failure_stages == set(STAGES) - {"prompt_assembly"}, ( "prompt assembly is the one stage with no recorded production failure, " "because it is pure string work under our control" )def test_healthy_request_passes_all_five_stages(): result = run_chain(load_recorded("healthy_billing_refund")) trace = result["trace"] assert trace.stages_seen() == list(STAGES) assert result["routed_queue"] == "billing" assert "billing specialists" in result["message"]def test_every_trace_event_is_plain_structured_json(): """No vendor SDK required. Structured log lines are the whole instrument.""" result = run_chain(load_recorded("healthy_billing_refund")) for line in result["trace"].as_log_lines(): decoded = json.loads(line) assert set(decoded) == {"stage", "summary", "duration_ms", "payload"} assert decoded["stage"] in STAGESdef test_retrieval_records_the_context_it_actually_found(): result = run_chain(load_recorded("healthy_billing_refund")) payload = result["trace"].payload_for("retrieval") assert payload["dominant_topic"] == "billing" assert [doc["doc_id"] for doc in payload["docs"]] == ["kb-101", "kb-102"]@pytest.mark.parametrize( "request_id", [ "empty_retrieval", "ambiguous_context", "malformed_model_output", "model_output_missing_tool_call", "misrouted_billing_to_technical", "unfilled_placeholder_in_message", ],)def test_failure_is_attributed_to_the_recorded_stage(request_id): """The assertion that fires is the one belonging to the stage that broke.""" recorded = load_recorded(request_id) expected_stage = recorded["expected_failure_stage"] with pytest.raises(StageAssertionError) as excinfo: run_chain(recorded) error = excinfo.value assert error.stage == expected_stage, ( f"{request_id} broke in {error.stage!r}, but the recorded break was {expected_stage!r}" ) # The stage that failed is the last one recorded, so everything before it is # proven good. That is what narrows a production investigation. assert error.trace.stages_seen()[-1] == expected_stage assert error.stage in str(error)def test_billing_misroute_localizes_to_the_tool_call_stage(capsys): """The worked example from the article. The retrieved context is unambiguously about billing, the model returns a perfectly well formed response, and the routing tool still picks the technical queue. Output level checking would say "wrong queue". The stage assertion says "the tool call stage chose technical while retrieval held billing", which is an actionable bug report. """ with pytest.raises(StageAssertionError) as excinfo: run_chain(load_recorded("misrouted_billing_to_technical")) error = excinfo.value assert error.stage == "tool_call" # Retrieval, prompt assembly and the model call all passed their assertions, # so none of them is the suspect. assert error.trace.stages_seen() == ["retrieval", "prompt_assembly", "model_call", "tool_call"] assert error.trace.payload_for("retrieval")["dominant_topic"] == "billing" assert error.trace.payload_for("model_call")["parse_error"] is None tool_payload = error.trace.payload_for("tool_call") assert tool_payload["arguments"]["queue"] == "technical" assert tool_payload["retrieved_topic"] == "billing" assert set(tool_payload["retrieved_doc_topics"]) == {"billing"} # Print the report so `pytest -v -s` shows the reader what a localized # failure looks like without needing the suite to go red. print("\n" + str(error)) assert "tool_call" in capsys.readouterr().out@pytest.mark.xfail( strict=True, reason=( "Deliberately failing case. Run with --runxfail to see pytest print the " "stage that broke and the payload it captured." ),)def test_deliberately_failing_case_prints_the_localized_stage(): """A real red test, kept xfail so a clean checkout still reports green. `pytest tests/ -v --runxfail` turns this into an honest failure and shows the localization mechanism working end to end. """ run_chain(load_recorded("misrouted_billing_to_technical"))
Run it against a real request and the assertion that fails tells you which stage broke, not just that the final output was wrong. That's the difference between an afternoon of debugging and five minutes: instead of staring at the final response wondering whether the problem was retrieval or the model, the log tells you the tool call stage picked route_to_queue("technical") when the retrieved context was unambiguously about billing, and the search narrows to one function. Getting that stage right before it ships is its own discipline, covered in testing AI agents that take actions. This tracing habit is the same discipline behind reliability work generally; we go deeper on what it takes to trust an agent once it's live in AI agent reliability testing.
Sampling and Scoring Live Traffic Instead of Guessing
One traced incident tells you what happened once. It doesn't tell you how often. Before you touch anything, whether that's a prompt, a model version, or a rollback, you need to know if the ticket that reached support is one weird outlier or the visible tenth of a pattern that's been running for two days.
The move is to pull a random sample of recent live requests, the last few hundred or few thousand, and score each one against a small set of expected criteria: did the response's claim match what was actually retrieved, did the tool call match the labeled-correct action, does the answer avoid contradicting the account state. Score with exact string matching and you'll manufacture false failures out of harmless rephrasing; two AI responses can say the same true thing in different words, and a strict match will flag both as broken. Score with semantic similarity or a small rubric instead, and a sample that scores consistently low across many requests, not just one, is a real signal. A single low score in an otherwise clean sample is noise; if you can't tell the difference between noise and a regression, the assertion is usually too strict, or the underlying prompt is ambiguous enough that even a person would answer it two different ways. That scoring problem is the same one that makes pre-production assertions hard, and the approaches carry over from testing non-deterministic AI outputs.
Here's a sampler that pulls logged request and response pairs, scores each with a similarity check instead of an exact match, and prints the aggregate pass rate along with the specific cases that fell below threshold for a human to look at:
"""Score a sample of logged production traffic without exact string matching.Run it: python src/eval_sampler.py # a healthy sample python src/eval_sampler.py --sample regressed # a real regressionThe sampler reads request and response pairs that were already logged, scoreseach response against one or more acceptable reference answers using semanticsimilarity (difflib, standard library only), prints an aggregate pass rate, andcompares that rate against a stored baseline.Three things the article insists on, encoded here:1. Exact string matching manufactures false failures. "Refunds land in five to seven business days" and "You should see the refund within a week" are the same answer. Similarity scoring accepts both, exact matching fails one.2. A single low score in an otherwise clean sample is noise. It gets flagged for a human to read, not treated as a broken build.3. A consistently low aggregate across many requests is the real signal. That is what the baseline comparison is for.No network calls, no API keys, no eval vendor SDK. Structured logs plus thestandard library are enough to run this on a laptop or in CI."""from __future__ import annotationsimport argparseimport jsonimport statisticsimport sysfrom difflib import SequenceMatcherfrom pathlib import Pathfrom typing import AnyFIXTURES = Path(__file__).resolve().parents[1] / "fixtures"SAMPLES = { "current": FIXTURES / "eval_sample_current.json", "regressed": FIXTURES / "eval_sample_regressed.json",}BASELINE = FIXTURES / "eval_baseline.json"def normalize(text: str) -> str: """Lowercase, drop punctuation, collapse whitespace. Rephrasing that only changes casing or punctuation should not cost a point. """ kept = "".join(ch.lower() if ch.isalnum() else " " for ch in text) return " ".join(kept.split())def similarity(response: str, reference: str) -> float: """Similarity in [0, 1] that tolerates rephrasing and reordering. Two views are compared and the more generous one wins: the normalized text as written (catches close paraphrase) and the same tokens sorted (catches an answer that says the same thing in a different order). """ left, right = normalize(response), normalize(reference) direct = SequenceMatcher(None, left, right).ratio() order_free = SequenceMatcher( None, " ".join(sorted(left.split())), " ".join(sorted(right.split())) ).ratio() return round(max(direct, order_free), 3)def score_case(case: dict[str, Any]) -> dict[str, Any]: """Score one logged response against every acceptable reference answer.""" references = case["acceptable_answers"] scored = sorted( ((similarity(case["response"], ref), ref) for ref in references), key=lambda pair: -pair[0], ) best_score, best_reference = scored[0] exact = any(normalize(case["response"]) == normalize(ref) for ref in references) return { "request_id": case["request_id"], "prompt": case["prompt"], "response": case["response"], "score": best_score, "closest_reference": best_reference, "exact_match": exact, "known_label": case.get("known_label", "unlabeled"), }def evaluate(sample: dict[str, Any]) -> list[dict[str, Any]]: return [score_case(case) for case in sample["cases"]]def truncate(text: str, width: int = 58) -> str: return text if len(text) <= width else text[: width - 3] + "..."def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument( "--sample", choices=sorted(SAMPLES), default="current", help="which logged sample to score (default: current)", ) parser.add_argument( "--threshold", type=float, default=None, help="similarity below which a case is flagged for human review (default: from the baseline file)", ) parser.add_argument( "--tolerance", type=float, default=None, help="how far the aggregate pass rate may fall below baseline before it counts as a regression (default: from the baseline file)", ) args = parser.parse_args(argv) with BASELINE.open(encoding="utf-8") as handle: baseline = json.load(handle) with SAMPLES[args.sample].open(encoding="utf-8") as handle: sample = json.load(handle) threshold = args.threshold if args.threshold is not None else baseline["threshold"] tolerance = args.tolerance if args.tolerance is not None else baseline["tolerance"] results = evaluate(sample) passed = [r for r in results if r["score"] >= threshold] flagged = [r for r in results if r["score"] < threshold] pass_rate = round(len(passed) / len(results), 3) exact_rate = round(sum(1 for r in results if r["exact_match"]) / len(results), 3) median_score = round(statistics.median(r["score"] for r in results), 3) print(f"sample: {args.sample} ({SAMPLES[args.sample].name})") print(f"window: {sample['window']}") print(f"cases: {len(results)} similarity threshold: {threshold}\n") print(f"{'request':<10} {'score':>6} {'verdict':<8} {'response (truncated)':<58}") print("-" * 86) for row in sorted(results, key=lambda r: r["score"]): verdict = "pass" if row["score"] >= threshold else "FLAG" print(f"{row['request_id']:<10} {row['score']:>6.3f} {verdict:<8} {truncate(row['response']):<58}") print() print(f"aggregate pass rate : {pass_rate:.3f} ({len(passed)}/{len(results)})") print(f"median similarity : {median_score:.3f}") print(f"baseline pass rate : {baseline['pass_rate']:.3f} (tolerance {tolerance:.2f})") print(f"exact match rate : {exact_rate:.3f} <- what a strict string check would have scored") floor = round(baseline["pass_rate"] - tolerance, 3) regression = pass_rate < floor print() if flagged: print(f"flagged for human review ({len(flagged)}):") for row in flagged: print(f" {row['request_id']} score {row['score']:.3f} label {row['known_label']}") print(f" prompt : {truncate(row['prompt'], 76)}") print(f" response : {truncate(row['response'], 76)}") print(f" closest : {truncate(row['closest_reference'], 76)}") else: print("nothing flagged for human review.") print() if regression: print( f"REGRESSION: pass rate {pass_rate:.3f} is below the baseline floor {floor:.3f}. " "Many requests degraded at once, which is signal, not noise. Investigate the " "retrieval, prompt and model call stages for this window." ) return 1 print( f"WITHIN BASELINE: pass rate {pass_rate:.3f} clears the floor {floor:.3f}. " f"{len(flagged)} flagged case(s) are noise at this sample size. Read them, do not " "page anyone. If a flagged case keeps recurring across windows, it stops being noise." ) return 0if __name__ == "__main__": sys.exit(main())
An aggregate pass rate that's dropped against last week's baseline, on a fresh random sample, is the difference between "a user hit an edge case" and "we have a live regression." That distinction decides everything that happens next.
How Autonoma Tests What Tracing and Eval Sampling Cannot See
Everything above answers a narrow question well: once something is wrong in production, where in the chain did it break, and how often is it happening. It never answers a different, earlier question: could this exact wrong action have been caught before the feature ever reached the user who filed the ticket. The refund confirmation from the opening of this piece would have traced cleanly. The retrieval step ran. The model call ran. The tool call executed. Every stage in the chain produced a syntactically valid result. The only thing wrong was that the action taken didn't match reality, and that's a category of failure a trace can only find after a user has already seen it.
That's the gap Autonoma is built to close, on the other side of the timeline from everything in this piece. It runs behavioral end-to-end tests against a real running build of the application before merge, so the same kind of wrong action we just spent two sections tracing and sampling in production gets caught as a failing test in a pull request instead of a support ticket, the same logic behind agent regression testing: most of what breaks live has already broken once before, and a pre-merge check would have caught it.
Both are real. Only one of them runs before a user ever sees the mistake.
How to Test AI Response Time: Where the Seconds Actually Went
Time to First Token vs Total Completion Time
A slow AI feature fails just as visibly as a wrong one, and it's worth debugging with the same rigor instead of a shrug and "the model is just slow today." The first thing to split apart is time-to-first-token versus total completion time. A chat interface that streams tokens can feel fast even at high total latency because the user sees output start appearing in under a second; the same total latency on a blocking call, where nothing renders until the whole response is ready, feels broken. If your feature can stream and the interface isn't taking advantage of it, that's often the actual bug, not the model.
The second thing is which layer actually ate the seconds. A single "the AI response took four seconds" number blends network time, a retrieval step hitting a vector store, the model call itself, and any tool calls the agent made along the way. Time each stage the same way you traced them for correctness, and the slow stage is usually one specific hop, a retrieval query with no index, a tool call to a downstream service with its own latency problem, not the model call everyone assumes is the culprit.
Why p95 and p99 Matter More Than the Average
Percentiles matter more than averages here. A mean latency of 1.2 seconds can hide a p95 of 6 seconds and a p99 of 14, and it's the p95 and p99 users complain about, not the mean. Set your timeout and retry budget against those percentiles, not the average, or your retries will kick in exactly when the system is already struggling. This is the same reasoning behind Google's SRE guidance on latency SLOs, where the tail, not the mean, defines the user experience.
Here's a latency checker that takes a set of recorded per-request timings, computes p50, p95, and p99 for both time-to-first-token and total completion, and flags any percentile that breaches a configurable budget:
"""Percentile latency check for an AI feature, per stage and end to end.Run it: python src/latency_check.py python src/latency_check.py --timeout-p99 20000 --retry-p95 6000It reads recorded per request timings (bundled fixture, standard library only)and computes p50, p95 and p99 for both time to first token and total completiontime, then breaks the same percentiles down per stage.Three things the article insists on, encoded here:1. Percentiles matter more than averages. The bundled window has a mean total of about 1.2 seconds, which hides a p95 near 6 seconds and a p99 of 14 seconds. The mean is the one number that tells you nothing is wrong.2. Timeout and retry budgets belong against p95 and p99, not the mean. A retry deadline set just above the average fires on a large slice of real traffic and doubles load on a system that is already struggling.3. Per stage timing localizes which layer ate the seconds. In this window it is not always the model: two of the slow requests are retrieval bound, and you would never see that from an end to end number alone.No network calls and no vendor exporter. Percentiles over structured log fieldsare enough."""from __future__ import annotationsimport argparseimport jsonimport mathimport statisticsimport sysfrom pathlib import Pathfrom typing import AnyFIXTURE = Path(__file__).resolve().parents[1] / "fixtures" / "latency_samples.json"PERCENTILES = (50, 95, 99)def percentile(values: list[float], p: float) -> float: """Nearest rank percentile, the convention latency dashboards use. Nearest rank always returns an observed measurement rather than an interpolated one, so a reported p99 is a request that really happened. """ if not values: raise ValueError("cannot take a percentile of an empty series") ordered = sorted(values) rank = max(1, math.ceil(p / 100 * len(ordered))) return float(ordered[rank - 1])def summarize(values: list[float]) -> dict[str, float]: return { "count": len(values), "mean": round(statistics.fmean(values), 1), "p50": percentile(values, 50), "p95": percentile(values, 95), "p99": percentile(values, 99), "max": float(max(values)), }def series(requests: list[dict[str, Any]], key: str) -> list[float]: return [float(r[key]) for r in requests]def stage_series(requests: list[dict[str, Any]]) -> dict[str, list[float]]: names = list(requests[0]["stages"]) return {name: [float(r["stages"][name]) for r in requests] for name in names}def print_row(label: str, stats: dict[str, float]) -> None: print( f"{label:<22} {stats['mean']:>10.1f} {stats['p50']:>10.1f} " f"{stats['p95']:>10.1f} {stats['p99']:>10.1f} {stats['max']:>10.1f}" )def main(argv: list[str] | None = None) -> int: with FIXTURE.open(encoding="utf-8") as handle: data = json.load(handle) defaults = data["budgets_ms"] parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--ttft-p95", type=float, default=defaults["ttft_p95"], help="time to first token budget, checked against p95 (ms)") parser.add_argument("--retry-p95", type=float, default=defaults["retry_p95"], help="retry deadline, checked against total p95 (ms)") parser.add_argument("--timeout-p99", type=float, default=defaults["timeout_p99"], help="hard request timeout, checked against total p99 (ms)") parser.add_argument("--stage-p99", type=float, default=defaults["stage_p99"], help="per stage budget, checked against each stage p99 (ms)") args = parser.parse_args(argv) requests = data["requests"] ttft = summarize(series(requests, "ttft_ms")) total = summarize(series(requests, "total_ms")) stages = {name: summarize(values) for name, values in stage_series(requests).items()} print(f"window: {len(requests)} requests from {FIXTURE.name}\n") header = f"{'series (ms)':<22} {'mean':>10} {'p50':>10} {'p95':>10} {'p99':>10} {'max':>10}" print(header) print("-" * len(header)) print_row("time to first token", ttft) print_row("total completion", total) print("-" * len(header)) for name, stats in stages.items(): print_row(f" {name}", stats) print() print("the mean is the number that lies: " f"{total['mean']:.0f} ms average against a {total['p95']:.0f} ms p95 " f"and a {total['p99']:.0f} ms p99.") breaches: list[str] = [] if ttft["p95"] > args.ttft_p95: breaches.append( f"time to first token p95 is {ttft['p95']:.0f} ms against a {args.ttft_p95:.0f} ms budget. " "One request in twenty waits this long before seeing a single character." ) if total["p95"] > args.retry_p95: breaches.append( f"total p95 is {total['p95']:.0f} ms against a {args.retry_p95:.0f} ms retry deadline. " "More than five percent of requests will be retried while the first attempt is still running, " "which adds load to whatever is already slow. Set the retry deadline above p95, not above the mean." ) if total["p99"] > args.timeout_p99: breaches.append( f"total p99 is {total['p99']:.0f} ms against a {args.timeout_p99:.0f} ms hard timeout. " "One request in a hundred is being killed mid flight and the user sees an error, not a slow answer." ) for name, stats in stages.items(): if stats["p99"] > args.stage_p99: breaches.append( f"stage {name} has a p99 of {stats['p99']:.0f} ms against a {args.stage_p99:.0f} ms per stage budget." ) print() if breaches: print(f"BUDGET BREACHES ({len(breaches)}):") for line in breaches: print(f" - {line}") else: print("no budget breaches in this window.") # Localize the tail: rank stages by how much of the p99 request each one owns, # so the fix lands on the layer that actually ate the seconds. worst = max(stages.items(), key=lambda item: item[1]["p99"]) print() print(f"tail localization: {worst[0]} owns the largest p99 at {worst[1]['p99']:.0f} ms.") slow_cutoff = total["p95"] slow = [r for r in requests if r["total_ms"] >= slow_cutoff] print(f"requests at or above the total p95 ({slow_cutoff:.0f} ms): {len(slow)}") for r in sorted(slow, key=lambda r: -r["total_ms"]): dominant = max(r["stages"].items(), key=lambda item: item[1]) print( f" {r['request_id']} total {r['total_ms']:>6} ms ttft {r['ttft_ms']:>6} ms " f"dominant stage {dominant[0]} ({dominant[1]} ms) note: {r['note']}" ) print() print("not every slow request is the model's fault. Read the dominant stage column " "before you go tuning prompts.") return 1 if breaches else 0if __name__ == "__main__": sys.exit(main())
Run this against a day's worth of logged timings, and a persistent p95 breach on one specific stage, not a one-off spike, is a real target for a fix, whether that's an index, a timeout tuned to the actual distribution, or a fallback path for the slow dependency.
Confirming the Regression Is Real, Then Rolling It Back
By this point you've traced the failing stage, sampled a set of live requests to see how often it's happening, and checked whether latency is part of the story. The last step is confidence: is this a real regression, and if so, what changed.
A regression is real when the sampled pass rate has genuinely dropped against a recent baseline, not one bad request in an otherwise clean sample, and when you can point to something that changed at roughly the same time, a prompt edit, a knowledge-base update, or a model version the provider swapped underneath the same API name with no changelog entry. If nothing you control changed, the model itself likely moved, which is exactly the silent-update pattern that shows up when you compare structured agent behavior over time rather than trusting a single anecdote.
Once you've confirmed it, rolling back is the boring, correct move, and boring is the point during an active incident. Revert the prompt to the last known-good version, pin the model to the version that was working if the provider's API allows it, or roll back the knowledge-base documents that changed. Don't try to patch the prompt live while the regression is still active; you'll be debugging your fix and the original problem at the same time, against sampled traffic that's still scoring low from both. Roll back first, confirm the sampled pass rate recovers, then make the deliberate fix as a separate, reviewed change. The deliberate fix that follows the rollback belongs in a pre-ship check, which is the workflow in how to QA a generative AI feature.
Turning Every Incident Into a Pre-Merge Check
Every incident you debug this way is also a finished specification, written by production at the price of a support ticket. The trace named the stage that produced the wrong action. The sampled pass rate said how often it happened. The rollback established what correct looked like. All three together describe a test case precisely enough to run on every future pull request, and the only question worth asking after the incident closes is whether anything actually runs it.
That is the loop Autonoma is built to close. It runs behavioral end-to-end tests against a real running build of your application in a preview environment on the pull request, driving the feature the way a user would rather than reading the model's output, so the wrong tool call and the unsupported claim show up as a failing check instead of a page. The distinction matters most for exactly the failures this piece has been chasing: the ones where every status code is 200 and the response reads fine.
The part that keeps this from going stale on an AI feature specifically is the Diffs Agent, which reads each pull request's diff and updates the test suite alongside the code. Prompts, retrieval logic, and tool definitions change far more often than the interface wrapped around them, and a suite pinned to last quarter's prompt is a slower version of no suite at all. That is also why a hand-written regression test for today's incident tends to decay: the behavior it encodes moves, and nobody notices until the next support ticket.
None of this retires tracing or eval sampling. Instrumented stages and scored samples are how you find out what is wrong once a feature is live, and every AI feature eventually needs both. They answer what broke and how often. Autonoma answers the question this whole piece has been working backward toward: whether it needed to reach a user at all. Instrument the stages, sample the traffic, and put Autonoma in front of both, so the next version of the refund that never happened is a red check on a pull request instead of a support ticket at 2am.
Frequently Asked Questions
Debugging an AI feature in production means tracing a live request through its full call chain, retrieval, prompt assembly, the model call, any tool calls, and response parsing, to find which stage produced the wrong behavior, then confirming the pattern against a sample of live traffic rather than a single report. Unlike a typical bug, there's usually no exception or failed status code: the request succeeded, and the failure is in what the feature decided to do or say.
Because every layer of the stack usually completes successfully. The retrieval step runs, the model returns a well-formed response, and any tool call executes without raising an exception. The failure lives in the content or the decision, a wrong tool called, a claim the retrieved context doesn't support, not in an error condition any standard monitoring or alerting is built to catch.
Instrument every stage of the call chain, retrieval, prompt assembly, model call, tool call, response parsing, with structured log events that capture enough of the input and output to tell whether that stage's result was well-formed, not just the final response. When something goes wrong, comparing the stage-by-stage trace against what a correct run looks like localizes the failure to one specific step instead of leaving you guessing across the whole chain.
Separate time-to-first-token from total completion time, since a streaming interface can feel fast even at higher total latency while a blocking call at the same latency feels broken. Time each stage of the pipeline individually to find which layer, retrieval, the model call, or a tool call, is actually consuming the seconds, and set timeout and retry budgets against p95 and p99 latency rather than the average, since those percentiles are what users actually experience as slow.
Score a random sample of recent live requests using semantic similarity rather than exact-string matching, since two correct responses can be worded differently, and compare the aggregate pass rate against a recent baseline. A single low-scoring request in an otherwise healthy sample is normal noise; a real regression shows up as the aggregate pass rate dropping consistently across the sample, usually alongside something that changed around the same time, a prompt edit, a knowledge base update, or a silent model version change on the provider's side.
Roll back first. Revert the prompt, pin the model to the previous version if your provider allows it, or roll back the knowledge base documents that changed, then confirm the sampled pass rate recovers. Trying to patch the prompt while the regression is still live means debugging your fix and the original problem at the same time against traffic that's still scoring low from both.
Tracing and eval sampling both run after a feature is already live, so a real user is still the one who finds the wrong action. The pre-merge equivalent is a behavioral end-to-end test that drives the running application the way a user would and asserts on what the app actually did, rather than on how the response reads, since the failures in this article all return 200 with plausible text. That's the layer Autonoma covers: it runs those tests against a real build of your application in a preview environment on the pull request, and its Diffs Agent updates the suite from each pull request's diff, which matters for an AI feature specifically because prompts, retrieval logic, and tool definitions change far more often than the interface wrapped around them.
