How to Build a RAG Evaluation Framework in 4 Metrics
Tom PiaggioCo-Founder at Autonoma
A RAG evaluation framework is the set of metrics, benchmark data, and pass/fail thresholds you run against a retrieval-augmented generation system to catch regressions before they reach users. For an app builder, that means four metrics computed on your own questions and documents, not the twenty-metric catalog built for ML research teams, wired into CI so a regression fails the build instead of sitting on a dashboard nobody checks.
The demo went fine. It always does. Feed the RAG feature the three questions everyone asks in a demo and it answers cleanly, cites the right page, looks finished. Then a real user asks something one degree off-script, and the answer comes back fluent, confident, and wrong: a citation pointing at a document that never said what the model just claimed it said. Someone tweaks the prompt, or swaps the embedding model, or bumps the chunk size, and ships the change anyway. Did that change make the RAG feature better or worse? Nobody on the team can answer that with a number. Just a shrug and "it seemed fine when I tried it."
That shrug is the whole problem this post exists to fix. Not by pointing you at a bigger metric catalog, you already have one open in another tab, but by narrowing it down to the four numbers that actually change a ship decision, showing the code that computes them, and walking through the part almost nobody covers: building a benchmark out of your own documents instead of someone else's leaderboard.
Why the Twenty-Metric Catalog Fails an App Builder
Search for RAG evaluation metrics and you land in one of a handful of comprehensive catalogs. Evidently's metric library, the docs behind Confident-AI's DeepEval, Weights & Biases's eval guides: all three are thorough, all three are correct, and all three are written for an ML team standing up a general-purpose evaluation pipeline across many models and many use cases. That's a different job from shipping one RAG feature into one product. Read those docs as an app builder and you walk away with a list of eighteen to twenty-something metrics and zero guidance on which four to actually gate a merge on.
Platform vendors make the same gap worse in a different direction. Databricks and similar tooling frame RAG evaluation around their own managed pipeline, which is fine advice if you're already on that stack and useless if you're not, because the tooling doesn't transfer. None of this is a knock on any of it. Evidently, DeepEval, and Weights & Biases all show up later in this post as the actual libraries doing the scoring. The problem is scope, not quality: a catalog built to cover every case leaves you to do the hard part, deciding which four matter for yours, entirely on your own.
Here's the answer for a RAG feature that retrieves documents and generates an answer from them: faithfulness, answer relevancy, context precision, and context recall. Everything else in the standard catalogs is either a variant of one of those four, or a metric that matters for a narrower case than most app builders have. Start there. Add a fifth metric only when a specific, observed failure in your own product demands it, never because a blog post listed it.
The Four RAG Evaluation Metrics That Matter (and What Each One Catches)
Split the four down the middle by which half of the pipeline they watch. Context precision and context recall are retrieval-side: they judge what your retriever handed to the generator, before a single token of the answer got written. Faithfulness and answer relevancy are generation-side: they judge what the model did with what it was given, after retrieval already happened. That split matters in practice, because a metric that flags a generation problem tells you nothing about whether the real fault sits in your retriever, and chasing the wrong half of the pipeline based on the wrong metric wastes an afternoon.
Context recall catches a retriever that missed the point of the question entirely. It's computed by checking whether the chunk that actually contains the ground-truth answer shows up anywhere in what the retriever returned. Score it low and no amount of prompt tuning on the generation side fixes anything, because the model was never handed the piece it needed. Context precision catches the opposite failure: the right chunk is in there, but it's buried under three or four irrelevant ones, forcing the generator to sort signal from noise instead of answering cleanly off a clean set. High recall paired with low precision describes a retriever that grabs everything vaguely related and lets the model sort it out, which works until an unlucky chunk ordering makes it not work. For the mechanics of actually computing precision and recall against your retriever's real output, how to test RAG retrieval goes a layer deeper into the retrieval step specifically.
Faithfulness catches a model inventing a fact that isn't anywhere in the chunks it actually received: the classic hallucination-on-top-of-retrieval failure, where the retriever did its job correctly and the generation step still made something up. Answer relevancy catches the failure faithfulness is structurally blind to: an answer that is completely grounded in the retrieved chunks, cites them accurately, and still doesn't answer the question the user actually asked. A technically faithful non-answer passes faithfulness and fails relevancy, which is exactly why gating on faithfulness alone lets an entire category of real user complaints straight through the gate.
The four metrics split cleanly by which half of the pipeline they're actually watching, which is also how to know which half to go fix.
Computing the Four Metrics With Ragas and DeepEval
You don't hand-roll claim decomposition and entailment checks for four metrics when two well-maintained libraries already do it. Ragas scores at the dataset level, a batch of question, answer, retrieved-context, and ground-truth-answer rows scored all at once, which is the shape you want once you have more than a handful of examples. Here's a small worked example, three rows, all four metrics, in about fifteen lines once the dataset is built:
"""Score a RAG system on the four metrics that matter, using Ragas.Ragas works at the DATASET level: you hand it a batch of rows and it returnsone score per metric across the whole batch. That is the shape you want onceyou have more than a handful of examples and you are asking "did this changemake the feature better or worse overall?"Run it: export OPENAI_API_KEY=sk-... python ragas_eval.pyEvery metric below is computed by an LLM judge, so the scores move a littlebetween runs on identical input. That is normal, not a bug, and it is exactlywhy the CI gate in tests/test_rag_gate.py gates on a repeated-run pass rateinstead of a single verdict."""import osfrom datasets import Datasetfrom ragas import evaluatefrom ragas.metrics import ( answer_relevancy, context_precision, context_recall, faithfulness,)# Ragas 0.2+ column names, and what each one has to contain:# user_input the question that was asked# response what your RAG pipeline actually answered# retrieved_contexts the chunks your retriever returned for that question# reference the human-written ground-truth answer## Faithfulness and answer relevancy only need the generation side.# Context precision and context recall need `reference` too, because they# compare what was retrieved against what the correct answer required.EXAMPLES = { "user_input": [ "When does my invoice actually get sent out?", "How many API calls can I make on the Team plan?", "Can I get a refund if I cancel three weeks in?", ], "response": [ ( "Meridian bills on the same calendar day you subscribed. The invoice " "is generated at 00:00 UTC on that day and emailed to your billing " "contact." ), ( "The Team plan allows 600 requests per minute. The limit is measured " "per workspace over a sliding 60 second window, not per API key." ), ( "A full refund is available within 14 days of your first paid invoice. " "Three weeks in, a monthly renewal is outside that window, so it is not " "refundable, though annual plans can still get a prorated refund within " "30 days." ), ], "retrieved_contexts": [ [ "Billing cycles and invoices: Meridian bills monthly on the calendar " "day you subscribed. Invoices are generated at 00:00 UTC on the billing " "day and emailed to the workspace billing contact.", "Annual plans are billed once for twelve months upfront and receive a " "15 percent discount against the monthly rate.", ], [ "API rate limits by plan: Free allows 60 requests per minute, Team " "allows 600 requests per minute, and Enterprise allows 6000 requests " "per minute. Limits apply per workspace, not per API key, and are " "measured over a sliding 60 second window.", ], [ "Refund policy: a full refund is available within 14 days of the first " "paid invoice. Annual plans are eligible for a prorated refund within " "30 days. Monthly renewals are not refundable after 14 days.", "Requesting a refund: email billing@meridian.example with the invoice " "number. Approved refunds are processed in 5 to 7 business days.", ], ], "reference": [ ( "Invoices are generated at 00:00 UTC on your monthly billing day, which " "is the calendar day you subscribed, and emailed to the billing contact." ), ( "The Team plan allows 600 requests per minute, applied per workspace " "over a sliding 60 second window." ), ( "A monthly renewal is not refundable after 14 days, so a cancellation at " "three weeks does not qualify. Annual plans remain eligible for a " "prorated refund within 30 days." ), ],}# The only four metrics an app builder needs. Two watch retrieval# (context precision, context recall), two watch generation# (faithfulness, answer relevancy).METRICS = [faithfulness, answer_relevancy, context_precision, context_recall]def main() -> None: if not os.getenv("OPENAI_API_KEY"): raise SystemExit( "OPENAI_API_KEY is not set. Every metric here is scored by an LLM " "judge, so the key is required.\n\n export OPENAI_API_KEY=sk-..." ) dataset = Dataset.from_dict(EXAMPLES) result = evaluate(dataset=dataset, metrics=METRICS) # to_pandas() gives one row per example and one column per metric, so you # can see which specific question dragged a score down instead of only # seeing the batch average. per_row = result.to_pandas() print(f"\nScored {len(per_row)} examples on {len(METRICS)} metrics\n") print(f"{'metric':<22}{'mean':>8}{'min':>8}") print("-" * 38) for metric in METRICS: column = per_row[metric.name] print(f"{metric.name:<22}{column.mean():>8.3f}{column.min():>8.3f}") print() # The minimum column is the one worth reading. A batch mean of 0.90 with a # single row at 0.20 is one broken question, not a uniformly decent system. worst = per_row[[m.name for m in METRICS]].min(axis=1).idxmin() print(f"Weakest example (row {worst}): {per_row.loc[worst, 'user_input']}\n")if __name__ == "__main__": main()
DeepEval scores per test case instead of per dataset, which is the shape you want when you're asserting on one scenario at a time inside a normal pytest run rather than scoring a batch offline. Same four metrics, same underlying idea, different unit of work:
"""Score a single RAG interaction on the same four metrics, using DeepEval.Where Ragas scores a dataset, DeepEval scores a TEST CASE. That is the shapeyou want when you are asserting on one scenario at a time inside a normalpytest run, rather than scoring a batch offline.Run it: export OPENAI_API_KEY=sk-... python deepeval_eval.pyThe four metrics map one-to-one onto the Ragas four: FaithfulnessMetric -> faithfulness AnswerRelevancyMetric -> answer_relevancy ContextualPrecisionMetric -> context_precision ContextualRecallMetric -> context_recallEach metric returns a score in [0, 1] plus a natural-language reason, which isthe part worth reading when a gate fails. A score tells you something broke.The reason tells you which claim was unsupported or which chunk was missing."""import osfrom deepeval.metrics import ( AnswerRelevancyMetric, ContextualPrecisionMetric, ContextualRecallMetric, FaithfulnessMetric,)from deepeval.test_case import LLMTestCase# An LLMTestCase carries the same four pieces of information a Ragas row does:# input the question that was asked# actual_output what your RAG pipeline answered# retrieval_context the chunks your retriever returned# expected_output the human-written ground-truth answer## Faithfulness and answer relevancy do not need expected_output. The two# contextual metrics do, because they judge retrieval against the answer the# question was supposed to get.TEST_CASE = LLMTestCase( input="What do I do when the API starts returning 429?", actual_output=( "A 429 means you exceeded your plan's rate limit. Read the Retry-After " "header, which gives the number of seconds to wait, and retry with " "exponential backoff. You can also watch X-RateLimit-Remaining to back " "off before you hit the ceiling. Note that bulk endpoints count as ten " "requests each." ), expected_output=( "Respect the Retry-After header on the 429 response, back off " "exponentially, and monitor X-RateLimit-Remaining and X-RateLimit-Reset " "to avoid hitting the limit at all. Bulk endpoints count as ten requests " "against the limit." ), retrieval_context=[ ( "Handling 429 responses: a 429 response includes a Retry-After header " "giving the number of seconds to wait before retrying. Clients should " "retry with exponential backoff. Every response also carries " "X-RateLimit-Remaining and X-RateLimit-Reset headers." ), ( "Bulk endpoints count as ten requests against your rate limit rather " "than one, because each call fans out server side." ), ( "API rate limits by plan: Free allows 60 requests per minute, Team " "allows 600 requests per minute, and Enterprise allows 6000 requests " "per minute." ), ],)# threshold= is DeepEval's own per-metric pass mark, and it is what makes# metric.is_successful() meaningful. These mirror the constants in# tests/test_rag_gate.py: faithfulness and retrieval recall are held higher,# relevancy and precision get a little more room.## Built inside a function rather than at import time on purpose: DeepEval# resolves its judge model in the metric constructor, so building these at# module scope turns a missing API key into an import error.def build_metrics() -> list: return [ FaithfulnessMetric(threshold=0.90, include_reason=True), AnswerRelevancyMetric(threshold=0.80, include_reason=True), ContextualPrecisionMetric(threshold=0.75, include_reason=True), ContextualRecallMetric(threshold=0.90, include_reason=True), ]def main() -> None: if not os.getenv("OPENAI_API_KEY"): raise SystemExit( "OPENAI_API_KEY is not set. Every metric here is scored by an LLM " "judge, so the key is required.\n\n export OPENAI_API_KEY=sk-..." ) print(f"\nQuestion: {TEST_CASE.input}\n") for metric in build_metrics(): # measure() runs the LLM judge synchronously and populates # metric.score, metric.reason, and metric.is_successful(). metric.measure(TEST_CASE) verdict = "PASS" if metric.is_successful() else "FAIL" print(f"{metric.__class__.__name__}") print(f" score : {metric.score:.3f} (threshold {metric.threshold:.2f}) {verdict}") print(f" reason : {metric.reason}\n")if __name__ == "__main__": main()
Both are computing the same thing under the hood: an LLM judge decomposing the answer into claims and checking each one against the retrieved chunks, or checking whether the ground-truth chunk shows up in what was retrieved. The choice between the two comes down to whether your workflow is dataset-first (Ragas) or test-case-first (DeepEval), not which one is more "correct." Pick one, wire it into the benchmark set below, and move on. Switching between them later costs you an afternoon, not a rewrite.
Building a RAG Benchmark Set From Your Own Documents
This is the part every metric catalog skips, and it's the part that actually decides whether your evaluation numbers mean anything. A metric computed against a public leaderboard or an academic benchmark tells you how a model performs on someone else's documents and someone else's questions. It tells you nothing about whether your users get good answers about your product's refund policy, your API's rate limits, or your onboarding flow, because none of those exist in any public dataset. The only benchmark that predicts whether your RAG feature works is one built from your own corpus and your own users' actual questions.
Start smaller than feels responsible. Thirty to fifty question-and-answer pairs, built from your real documents, beats a hundred you're still debating six weeks from now, and it beats zero while you wait for a "proper" dataset that never quite gets prioritized. A benchmark that small already catches the regressions that matter: did the last prompt change break the three questions that come up in every sales call, did swapping embedding models drop recall on the ten questions your top customers actually ask. You extend it as real gaps show up, not before.
Source the questions from where your users already are, not from your own imagination. Support tickets, sales call transcripts, and product chat logs are full of exactly the phrasing real users use, which is reliably weirder, shorter, and more ambiguous than the clean questions a team invents while staring at its own documentation. Mining a week of support tickets for lines that end in a question mark, then de-duplicating and skimming the results, produces a stronger seed set in an afternoon than a team brainstorming session produces in a week, because it's grounded in what people actually asked instead of what a team assumes they'd ask.
Writing the ground-truth answer is the part that takes the most care and the least code. For each sourced question, find the actual chunk (or chunks) in your corpus that contain the answer, write the correct answer in your own words, and record which chunk IDs support it. That chunk-ID record is what makes context precision and recall computable later; skip it and you can still score faithfulness and answer relevancy, but you lose the retrieval-side half of the evaluation entirely.
Your corpus changes, so the benchmark has to version alongside it, not sit frozen next to a wiki page from three months ago. Commit the benchmark file into the same repository as your RAG pipeline code, bump its version when a source document it depends on is edited or removed, and treat a stale ground-truth answer (correct against last quarter's docs, wrong against this quarter's) as a bug in the benchmark, not a mysterious drop in your metric scores.
That versioning discipline is manual by necessity here, because only a human knows what the corpus is supposed to say. It's worth naming what the equivalent looks like when the artifact is derived from code instead of documents, since it's the same rot with a different fix: on the behavioral side, Autonoma's Diffs Agent reads each pull request's diff and adds, updates, or deprecates the end-to-end cases that change affects, so the suite tracks the application without a scheduled review. Your ground-truth answers still need your judgment. The cases that assert on what the app did with those answers don't.
The benchmark that matters is the one built from what your users actually asked about your actual documents, not a public leaderboard.
Here's a starting point for mining candidate questions out of exported support logs, heuristic-based, meant for a human to review before anything becomes a graded benchmark entry:
#!/usr/bin/env python3"""Mine candidate benchmark questions out of exported support logs. python scripts/extract_questions.py --input logs/ --output candidates.jsonlStandard library only, on purpose. This is a grep with opinions, not a model.WHAT THIS PRODUCES IS CANDIDATES, NOT TRUTH.============================================Every line written to the output file is a *suggestion for a human to review*.Nothing here is a graded benchmark entry, and nothing here has a ground-truthanswer, because a ground-truth answer requires a person who knows the productto read the source documents and write the correct answer down. Feeding thisoutput straight into an evaluation run would score your RAG system againstquestions nobody verified are answerable, which is worse than not measuring atall: it produces a number that looks rigorous and means nothing.The workflow this is built for: 1. Run this over a week of exported tickets or chat logs. 2. Skim the JSONL. Most of it is noise. Delete the noise. 3. For each keeper, find the chunk(s) in your corpus that answer it and write the ground-truth answer yourself. 4. Promote the survivors into benchmark/rag_benchmark_vN.jsonl.Steps 2 and 3 are the whole job. This script only saves you step 1."""from __future__ import annotationsimport argparseimport jsonimport reimport sysimport unicodedatafrom pathlib import Path# Files worth scanning. Anything else in the folder is skipped silently.TEXT_SUFFIXES = {".txt", ".md", ".log", ".csv", ".jsonl", ".json"}# A question shorter than this is almost always a fragment ("what?", "really?")# or an agent's own clarifier ("this one?"). Raise it if your logs are noisy.MIN_QUESTION_CHARS = 20MAX_QUESTION_CHARS = 300# Conversational scaffolding that shows up at the front of real support lines.# Stripped before the length check so "hi there, how do I rotate an api key?"# is measured on the part that matters.GREETING_PREFIXES = re.compile( r"^(?:" r"hi+|hey+|hello|good\s+(?:morning|afternoon|evening)|" r"thanks?|thank\s+you|sorry|excuse\s+me|quick\s+(?:one|question)|" r"just\s+wondering|hope\s+(?:you|this)\s+\w+|" r"also|and|so|oh|actually|btw|by\s+the\s+way" r")\b[\s,.!:-]*", re.IGNORECASE,)# Transcript line prefixes: "Customer:", "[14:03] user >", "2026-03-11 agent:".SPEAKER_PREFIX = re.compile( r"^\s*(?:\[?\d{1,4}[-:/]\d{1,2}(?:[-:/]\d{1,4})?\]?\s*)?" r"(?:\[?\d{1,2}:\d{2}(?::\d{2})?\]?\s*)?" r"(?:(?P<speaker>[A-Za-z][\w .'-]{0,30})\s*[:>]\s*)?",)# Lines the SUPPORT AGENT asked, not the user. Keeping these poisons the# benchmark with internal phrasing your users never use.AGENT_SPEAKERS = { "agent", "support", "admin", "bot", "assistant", "system", "operator", "staff", "moderator", "help", "helpdesk",}# Boilerplate questions that appear in every ticket and answer nothing about# the product.BOILERPLATE = re.compile( r"^(?:" r"(?:is\s+)?(?:there\s+)?anything\s+else|" r"(?:does\s+)?that\s+(?:help|make\s+sense|work)|" r"(?:are\s+)?you\s+(?:still\s+)?there|" r"can\s+you\s+hear\s+me|" r"how\s+are\s+you|" r"(?:may|can)\s+i\s+(?:have|get)\s+your\s+(?:email|name|order|account)" r")", re.IGNORECASE,)# Anything that looks like a secret or a personal identifier. A candidate# carrying one gets dropped rather than written to a file that ends up in git.## The credential branch allows underscores and dashes inside the token, so# "sk_live_abcdefgh12345678" is caught and not just "sk_abcdefgh12345678", and# it requires at least one digit in the tail. That digit requirement is what# keeps ordinary prose like "the key_management_system" from being flagged.# When in doubt this drops the candidate: losing one reviewable question costs# nothing, and writing a live key into a committed file costs a rotation.SENSITIVE = re.compile( r"(?:" r"[\w.+-]+@[\w-]+\.[\w.]+" # email address r"|\b(?:sk|pk|api|key|token|bearer)[-_](?=[A-Za-z0-9_-]*\d)[A-Za-z0-9_-]{8,}" r"|\b(?:\d[ -]?){13,19}\b" # card-length digit run r")", re.IGNORECASE,)def iter_lines(root: Path): """Yield (path, line_number, text) for every text-ish line under root. JSON and JSONL files get their string values pulled out, since exported chat logs are usually a list of message objects rather than raw text. """ paths = sorted(root.rglob("*")) if root.is_dir() else [root] for path in paths: if not path.is_file() or path.suffix.lower() not in TEXT_SUFFIXES: continue try: raw = path.read_text(encoding="utf-8", errors="replace") except OSError as exc: print(f"skipping {path}: {exc}", file=sys.stderr) continue if path.suffix.lower() in {".json", ".jsonl"}: for lineno, text in enumerate(_iter_json_strings(raw), start=1): yield path, lineno, text else: for lineno, text in enumerate(raw.splitlines(), start=1): yield path, lineno, textdef _iter_json_strings(raw: str): """Pull every string leaf out of a JSON or JSONL blob, in document order.""" def walk(node): if isinstance(node, str): yield node elif isinstance(node, dict): for value in node.values(): yield from walk(value) elif isinstance(node, list): for value in node: yield from walk(value) try: yield from walk(json.loads(raw)) return except json.JSONDecodeError: pass for line in raw.splitlines(): line = line.strip() if not line: continue try: yield from walk(json.loads(line)) except json.JSONDecodeError: yield linedef extract_question( line: str, min_chars: int = MIN_QUESTION_CHARS,) -> tuple[str, str | None] | None: """Return (question, speaker) if this line looks like a user question. Returns None for everything that fails a heuristic. Being aggressive here is correct: a human still reviews whatever survives, and a smaller pile with less junk in it actually gets reviewed. """ match = SPEAKER_PREFIX.match(line) speaker = (match.group("speaker") or "").strip().lower() if match else "" body = line[match.end():] if match else line # Drop the agent's own questions. if speaker and speaker.rstrip("s") in AGENT_SPEAKERS: return None body = GREETING_PREFIXES.sub("", body.strip()) # A single line can contain several sentences. Take the last one that is a # question, which in support text is almost always the actual ask. sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+", body) if s.strip()] question = next((s for s in reversed(sentences) if s.endswith("?")), None) if question is None: return None question = GREETING_PREFIXES.sub("", question).strip() if not (min_chars <= len(question) <= MAX_QUESTION_CHARS): return None if BOILERPLATE.match(question): return None if SENSITIVE.search(question): return None # Needs at least four words to be a real question rather than a fragment. if len(question.split()) < 4: return None return question, (speaker or None)def normalize(question: str) -> str: """Fold casing, whitespace, punctuation, and accents for de-duplication. "How do I rotate an API key?" and "how do i rotate an api key ??" are the same question and should only be reviewed once. """ folded = unicodedata.normalize("NFKD", question) folded = "".join(c for c in folded if not unicodedata.combining(c)) folded = folded.casefold() folded = re.sub(r"[^\w\s]", " ", folded) return re.sub(r"\s+", " ", folded).strip()def main() -> int: parser = argparse.ArgumentParser( description=( "Mine candidate benchmark questions from exported support tickets " "or chat logs. Output is CANDIDATES for human review, never " "auto-graded ground truth." ) ) parser.add_argument( "--input", required=True, type=Path, help="File or folder of exported logs to scan (recursive).", ) parser.add_argument( "--output", required=True, type=Path, help="Where to write the JSONL candidates.", ) parser.add_argument( "--min-chars", type=int, default=MIN_QUESTION_CHARS, help=f"Minimum question length in characters (default {MIN_QUESTION_CHARS}).", ) parser.add_argument( "--limit", type=int, default=0, help="Stop after N unique candidates. 0 means no limit.", ) args = parser.parse_args() if not args.input.exists(): print(f"error: input path does not exist: {args.input}", file=sys.stderr) return 1 seen: dict[str, dict] = {} scanned = 0 for path, lineno, line in iter_lines(args.input): scanned += 1 found = extract_question(line, min_chars=args.min_chars) if found is None: continue question, speaker = found key = normalize(question) if key in seen: # Already have this question. Record that it came up again, which # is the single most useful signal for deciding what to review first. seen[key]["occurrences"] += 1 continue seen[key] = { "question": question, "normalized": key, "occurrences": 1, "speaker": speaker, "source_file": str(path), "source_line": lineno, # Deliberately empty. A human fills these in during review. "ground_truth_answer": None, "ground_truth_chunk_ids": [], "reviewed": False, } if args.limit and len(seen) >= args.limit: break # Most-repeated questions first, so the reviewer's first ten minutes go to # whatever users actually keep asking. candidates = sorted(seen.values(), key=lambda c: (-c["occurrences"], c["question"])) args.output.parent.mkdir(parents=True, exist_ok=True) with args.output.open("w", encoding="utf-8") as handle: for candidate in candidates: handle.write(json.dumps(candidate, ensure_ascii=False) + "\n") print(f"scanned {scanned} lines under {args.input}") print(f"wrote {len(candidates)} unique candidates to {args.output}") print("These are CANDIDATES. Review them, write the ground truth by hand,") print("then promote the keepers into benchmark/rag_benchmark_vN.jsonl.") return 0if __name__ == "__main__": raise SystemExit(main())
And here's the shape a versioned benchmark entry actually takes once a human has reviewed and written the ground truth for it:
{"id": "bench-001", "question": "when does my invoice actually get sent out?", "ground_truth_answer": "Invoices are generated at 00:00 UTC on your monthly billing day, which is the calendar day you originally subscribed, and emailed to the workspace billing contact.", "ground_truth_chunk_ids": ["kb-billing-001"], "source": "support-ticket-4821"}{"id": "bench-002", "question": "if i upgrade halfway through the month do i get charged twice", "ground_truth_answer": "No. An upgrade takes effect immediately and you are charged a single prorated amount covering the remainder of the current billing period, not a second full invoice.", "ground_truth_chunk_ids": ["kb-billing-002", "kb-billing-001"], "source": "support-ticket-4903"}{"id": "bench-003", "question": "how many api calls can I make on the team plan?", "ground_truth_answer": "The Team plan allows 600 requests per minute. The limit applies per workspace rather than per API key, and is measured over a sliding 60 second window.", "ground_truth_chunk_ids": ["kb-ratelimit-001"], "source": "product-chat-log-2026-03-04"}{"id": "bench-004", "question": "what should our client do when it starts getting 429s?", "ground_truth_answer": "Read the Retry-After header on the 429 response for the number of seconds to wait, then retry with exponential backoff. Watch X-RateLimit-Remaining and X-RateLimit-Reset to back off before hitting the limit, and remember that bulk endpoints count as ten requests each.", "ground_truth_chunk_ids": ["kb-ratelimit-002", "kb-ratelimit-001"], "source": "sales-call-2026-03-11"}{"id": "bench-005", "question": "how long do team invites stay valid before they expire?", "ground_truth_answer": "An invitation email is valid for 14 days. After that it expires and an Admin has to resend it.", "ground_truth_chunk_ids": ["kb-onboarding-001"], "source": "support-ticket-5017"}{"id": "bench-006", "question": "can i get my money back if i cancel after 3 weeks", "ground_truth_answer": "Not on a monthly renewal. Full refunds are only available within 14 days of the first paid invoice, so a cancellation at three weeks falls outside that window. Annual plans are still eligible for a prorated refund within 30 days of purchase, requested by emailing billing with the invoice number.", "ground_truth_chunk_ids": ["kb-refunds-001", "kb-refunds-002"], "source": "support-ticket-5122"}{"id": "bench-007", "question": "do we need to be on enterprise to log in with okta?", "ground_truth_answer": "Yes. SAML single sign-on, including Okta, is available on the Enterprise plan only. It is configured by a workspace Admin under Settings, Security, Single sign-on.", "ground_truth_chunk_ids": ["kb-sso-001"], "source": "sales-call-2026-02-27"}{"id": "bench-008", "question": "what happens to our workspace if the card on file gets declined?", "ground_truth_answer": "Meridian retries the charge three times over seven days and emails the billing contact after each attempt. If all three retries fail, the workspace becomes read-only until a valid payment method is added, and data is retained for 30 days after that point.", "ground_truth_chunk_ids": ["kb-billing-003"], "source": "support-ticket-5188"}
Turning Your RAG Evaluation Framework Into a Pass/Fail CI Gate
A metric score that lives on a dashboard nobody opens is not an evaluation framework, it's a chart. The whole point of the four metrics and the benchmark set is to fail a build automatically when a change makes the RAG feature worse, the same way a broken unit test fails a build, without anyone having to remember to go look.
Set thresholds per metric, not one aggregate score for all four blended together. An aggregate hides exactly the information you need: a feature that's 95% faithful and 60% relevant to the question asked can average out to a number that looks fine while actively failing users on relevancy. Gate faithfulness and context recall higher and stricter, since an invented fact or a missed chunk is usually the worse failure to ship, and give answer relevancy and context precision a little more room, since a slightly over-broad retrieval or a technically-adjacent answer is a rougher edge, not a wrong one.
The judge scoring your metrics is itself a non-deterministic model, and pretending otherwise is how a gate becomes untrustworthy within a month. Run each benchmark question through the pipeline and the judge multiple times, not once, and gate on a pass rate across those runs rather than a single verdict. Four-of-five passing is a defensible floor for most teams; a question that drops from five-of-five to two-of-five over a week is a real signal, not noise, the kind a single-run gate would never surface. A gate that fails on normal judge variance gets disabled by the team within a sprint, and a disabled gate is worse than no gate, because everyone still believes it's running.
"""The pass/fail CI gate. This is the file the whole repo exists for. pytest tests/test_rag_gate.pyWhat it does, in order: 1. Loads the versioned benchmark (benchmark/rag_benchmark_v3.jsonl). 2. Runs YOUR RAG pipeline over every benchmark question. 3. Scores the results with Ragas on all four metrics. 4. Repeats steps 2 and 3 `GATE_RUNS` times. 5. Asserts each metric cleared its own threshold in at least `REQUIRED_PASSING_RUNS` of those runs.Two design decisions carry the whole thing.FIRST: a separate threshold per metric, never one blended average. A systemthat is 95 percent faithful and 60 percent relevant averages out to somethingthat looks acceptable while actively failing users on relevancy. Averagingdestroys the one piece of information you needed.SECOND: gate on a pass RATE across repeated runs, not a single verdict. Thejudge scoring these metrics is itself a non-deterministic LLM. A gate thatfails on ordinary judge variance gets disabled by the team inside a sprint,and a disabled gate is worse than no gate because everyone still believes itis running. Four passes out of five is a defensible floor. A question thatdrifts from five-of-five to two-of-five over a week is real signal, and it issignal a single-run gate can never surface."""from __future__ import annotationsimport importlibimport jsonimport mathimport osfrom dataclasses import dataclassfrom pathlib import Pathimport pytestfrom datasets import Datasetfrom ragas import evaluatefrom ragas.metrics import ( answer_relevancy, context_precision, context_recall, faithfulness,)# ---------------------------------------------------------------------------# Thresholds. Tune these; they are the dial the whole gate turns on.## Faithfulness and context recall sit higher because their failures are the# ones you genuinely cannot ship: an invented fact stated confidently, or a# missing chunk that made the correct answer unreachable in the first place.## Answer relevancy and context precision get more room because their failures# are rough edges rather than wrong answers: an over-broad retrieval that# dragged in a neighbouring chunk, or an answer that wandered one sentence# past what was asked.## Start here, watch a week of real runs, then move them. Do not move them# because a build went red once.# ---------------------------------------------------------------------------FAITHFULNESS_THRESHOLD = 0.90ANSWER_RELEVANCY_THRESHOLD = 0.80CONTEXT_PRECISION_THRESHOLD = 0.75CONTEXT_RECALL_THRESHOLD = 0.90# Repeated-run gate. Five runs, four of which must clear the threshold.# Override GATE_RUNS down to 1 for a fast local smoke check; leave it at 5 in CI.GATE_RUNS = int(os.getenv("RAG_GATE_RUNS", "5"))REQUIRED_PASSING_RUNS = int(os.getenv("RAG_GATE_REQUIRED_PASSES", "4"))BENCHMARK_PATH = Path( os.getenv("RAG_BENCHMARK", Path(__file__).parent.parent / "benchmark" / "rag_benchmark_v3.jsonl"))# ---------------------------------------------------------------------------# THE SEAM: point this at your own RAG pipeline.## Set RAG_PIPELINE to "your_module:your_callable". The callable takes a single# question string and returns anything exposing:## .answer -> str the generated answer# .contexts -> list[str] the chunk TEXTS your retriever returned## A plain dict with "answer" and "contexts" keys works too, as does a tuple of# (answer, contexts). Nothing else in this file needs to change.## export RAG_PIPELINE="myapp.rag:answer_question"# pytest tests/test_rag_gate.py## The default below is the throwaway reference pipeline in rag_pipeline.py,# there only so this gate runs before you have wired anything up.# ---------------------------------------------------------------------------PIPELINE_TARGET = os.getenv("RAG_PIPELINE", "rag_pipeline:run_rag_pipeline")def load_pipeline(): """Import the configured pipeline callable from its "module:function" string.""" if ":" not in PIPELINE_TARGET: raise ValueError( f'RAG_PIPELINE must look like "module:callable", got {PIPELINE_TARGET!r}' ) module_name, attr = PIPELINE_TARGET.split(":", 1) module = importlib.import_module(module_name) pipeline = getattr(module, attr, None) if not callable(pipeline): raise TypeError(f"{PIPELINE_TARGET} is not callable") return pipelinedef unpack(result) -> tuple[str, list[str]]: """Normalize whatever the pipeline returned into (answer, contexts).""" if hasattr(result, "answer") and hasattr(result, "contexts"): return result.answer, list(result.contexts) if isinstance(result, dict): return result["answer"], list(result["contexts"]) if isinstance(result, (tuple, list)) and len(result) == 2: return result[0], list(result[1]) raise TypeError( "Pipeline must return an object with .answer/.contexts, a dict with " f"those keys, or an (answer, contexts) pair. Got {type(result).__name__}." )def load_benchmark() -> list[dict]: """Read the versioned benchmark and fail loudly on a malformed entry.""" if not BENCHMARK_PATH.exists(): raise FileNotFoundError(f"Benchmark not found at {BENCHMARK_PATH}") required = {"id", "question", "ground_truth_answer", "ground_truth_chunk_ids"} entries = [] with BENCHMARK_PATH.open(encoding="utf-8") as handle: for lineno, line in enumerate(handle, start=1): line = line.strip() if not line: continue entry = json.loads(line) missing = required - entry.keys() if missing: raise ValueError( f"{BENCHMARK_PATH.name}:{lineno} is missing {sorted(missing)}. " "A benchmark entry without ground truth cannot be scored." ) entries.append(entry) if not entries: raise ValueError(f"{BENCHMARK_PATH} is empty. Nothing to gate on.") return entries# (metric object, threshold, human label). Indexing the Ragas result by# `metric.name` rather than a hardcoded string keeps this working across Ragas# versions, which rename some metric columns.GATED_METRICS = [ (faithfulness, FAITHFULNESS_THRESHOLD, "faithfulness"), (answer_relevancy, ANSWER_RELEVANCY_THRESHOLD, "answer relevancy"), (context_precision, CONTEXT_PRECISION_THRESHOLD, "context precision"), (context_recall, CONTEXT_RECALL_THRESHOLD, "context recall"),]@dataclassclass RunScores: """One full pass over the benchmark: mean score per metric.""" run_index: int scores: dict[str, float]def score_once(run_index: int, pipeline, benchmark: list[dict]) -> RunScores: """Run the pipeline over the whole benchmark and score it with Ragas once.""" questions, answers, contexts, references = [], [], [], [] for entry in benchmark: answer, retrieved = unpack(pipeline(entry["question"])) questions.append(entry["question"]) answers.append(answer) # Ragas needs a non-empty context list; an empty retrieval is a real # failure and should score zero rather than crash the run. contexts.append(retrieved or [""]) references.append(entry["ground_truth_answer"]) dataset = Dataset.from_dict( { "user_input": questions, "response": answers, "retrieved_contexts": contexts, "reference": references, } ) frame = evaluate( dataset=dataset, metrics=[metric for metric, _, _ in GATED_METRICS], ).to_pandas() return RunScores( run_index=run_index, scores={metric.name: float(frame[metric.name].mean()) for metric, _, _ in GATED_METRICS}, )@pytest.fixture(scope="session")def gate_runs() -> list[RunScores]: """Score the benchmark GATE_RUNS times. Session-scoped: this is the expensive part. Every gated metric reads from this one fixture, so the benchmark is scored GATE_RUNS times total, not GATE_RUNS times per metric. """ if not os.getenv("OPENAI_API_KEY"): pytest.skip("OPENAI_API_KEY is not set; the LLM judge cannot run.") pipeline = load_pipeline() benchmark = load_benchmark() results = [score_once(i, pipeline, benchmark) for i in range(1, GATE_RUNS + 1)] print(f"\nBenchmark: {BENCHMARK_PATH.name} ({len(benchmark)} questions)") print(f"Pipeline : {PIPELINE_TARGET}") print(f"Runs : {GATE_RUNS}, need {REQUIRED_PASSING_RUNS} passing\n") return results@pytest.mark.parametrize( ("metric", "threshold", "label"), GATED_METRICS, ids=[label.replace(" ", "_") for _, _, label in GATED_METRICS],)def test_metric_clears_threshold_across_runs(gate_runs, metric, threshold, label): """Each metric must clear its own threshold in at least N of M runs.""" observed = [run.scores[metric.name] for run in gate_runs] # A NaN means the judge failed to produce a score for that run. Counting it # as a pass would let a broken judge wave a regression through. passing = [s for s in observed if not math.isnan(s) and s >= threshold] formatted = ", ".join("nan" if math.isnan(s) else f"{s:.3f}" for s in observed) print( f"{label:<20} threshold {threshold:.2f} " f"runs [{formatted}] passed {len(passing)}/{len(observed)}" ) assert len(passing) >= REQUIRED_PASSING_RUNS, ( f"{label} cleared {threshold:.2f} in only {len(passing)} of {len(observed)} runs " f"(needed {REQUIRED_PASSING_RUNS}). Per-run means: [{formatted}]. " f"{'Consistently below threshold, so this is a real regression, not judge noise.' if not passing else 'Borderline: check whether the drop is a regression or a threshold set too tight for this benchmark.'}" )def test_retrieval_hits_ground_truth_chunks(): """A free, deterministic retrieval check that runs with no LLM at all. Context recall is the LLM-judged version of this question. But because the benchmark records `ground_truth_chunk_ids`, you can catch an outright broken retriever in milliseconds and for zero dollars, before spending anything on judges. Run this one on every commit and the expensive gate above only on pull requests. Point RAG_RETRIEVER at a retrieval-only callable that takes a question and returns chunks (dicts with a "chunk_id" key, objects with a .chunk_id attribute, or plain ID strings). Skipped if it is not importable. """ target = os.getenv("RAG_RETRIEVER", "rag_pipeline:retrieve") module_name, _, attr = target.partition(":") try: retriever = getattr(importlib.import_module(module_name), attr) except (ImportError, AttributeError): pytest.skip( f"No retrieval-only callable at {target}; the context recall gate covers this." ) def chunk_id_of(chunk): if isinstance(chunk, str): return chunk if isinstance(chunk, dict): return chunk.get("chunk_id") return getattr(chunk, "chunk_id", None) misses = [] for entry in load_benchmark(): retrieved = [chunk_id_of(c) for c in retriever(entry["question"])] expected = set(entry["ground_truth_chunk_ids"]) # At least one ground-truth chunk must show up. Requiring all of them # is a stricter gate worth turning on once retrieval is solid. if expected and not (expected & set(retrieved)): misses.append(f" {entry['id']}: expected any of {sorted(expected)}, got {retrieved}") assert not misses, ( f"Retrieval returned none of the ground-truth chunks for {len(misses)} " "benchmark questions:\n" + "\n".join(misses) )
Wire that gate into the pull request path itself, not a script someone is supposed to remember to run before a release:
name: RAG Eval Gate# Runs the four-metric gate on every pull request. A failing run blocks the# merge, which is the entire point: a metric that only lands on a dashboard# is a chart, not a gate.## Setup, once:# 1. Add OPENAI_API_KEY under Settings, Secrets and variables, Actions.# 2. Settings, Branches, add a branch protection rule on your default branch# and mark "rag-eval-gate / gate" as a required status check. Without# that step the job reports red but the merge button stays green.on: pull_request: branches: [main] # Nightly catch for drift that has nothing to do with your code: a provider # silently updating the model behind an alias will move these scores with # zero commits on your side. schedule: - cron: "0 6 * * *" workflow_dispatch:# A new push supersedes the in-flight run for the same PR. LLM judge calls# cost money; do not pay twice for a stale commit.concurrency: group: rag-eval-gate-${{ github.ref }} cancel-in-progress: truepermissions: contents: readjobs: gate: name: gate runs-on: ubuntu-latest # Five scored runs over the benchmark is real wall-clock time. Fail the # job rather than letting a hung judge call burn a runner for six hours. timeout-minutes: 30 steps: - name: Check out the repository uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.11" cache: pip cache-dependency-path: requirements.txt - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt # Free and instant. If retrieval is outright broken there is no reason # to spend anything on LLM judges, so this runs first and short-circuits. - name: Retrieval sanity check (no LLM calls) run: pytest tests/test_rag_gate.py::test_retrieval_hits_ground_truth_chunks -v - name: Four-metric eval gate env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} # Five runs, four of which must clear each per-metric threshold. # The repeated runs absorb LLM judge non-determinism so the gate # does not go red on ordinary variance. RAG_GATE_RUNS: "5" RAG_GATE_REQUIRED_PASSES: "4" # Swap in your own pipeline here, or set it as a repository variable. # RAG_PIPELINE: myapp.rag:answer_question run: pytest tests/test_rag_gate.py -v --tb=short - name: Explain a failure if: failure() run: | echo "### RAG eval gate failed" >> "$GITHUB_STEP_SUMMARY" echo "" >> "$GITHUB_STEP_SUMMARY" echo "Per-run metric means are printed in the job log above." >> "$GITHUB_STEP_SUMMARY" echo "" >> "$GITHUB_STEP_SUMMARY" echo "- All five runs well below the threshold: a real regression. Look at what changed in the prompt, the chunking, or the embedding model." >> "$GITHUB_STEP_SUMMARY" echo "- Three or four runs hovering right at the threshold: judge variance against a threshold set too tight for this benchmark. Widen it deliberately, in a commit, with a reason." >> "$GITHUB_STEP_SUMMARY" echo "- Faithfulness down but context recall steady: generation is inventing detail the retrieved chunks do not support." >> "$GITHUB_STEP_SUMMARY" echo "- Context recall down: retrieval stopped finding the chunk that holds the answer, so no prompt change will fix it." >> "$GITHUB_STEP_SUMMARY"
None of these four metrics, however carefully thresholded, tell you whether the citation link the model just generated actually opens the right document when a real user clicks it, or whether a grounded, relevant answer got wired into the right field on the next screen. Verifying what your application actually does with a response that already passed every metric is a behavioral layer above eval scores, and it's closer to what Autonoma tests for inside a running app than anything Ragas or DeepEval measure.
Shipping a RAG Feature You Can Actually Trust
Four metrics, split cleanly across retrieval and generation. A benchmark built from your own documents and your own users' real questions, thirty to fifty entries to start, versioned alongside the code instead of frozen next to a stale wiki page. Thresholds set per metric instead of blended into one misleading average, gated on a repeated-run pass rate instead of a single lucky or unlucky judge call. That's the whole RAG evaluation framework. It fits in a CI job, it runs on every pull request, and it turns "it seemed fine when I tried it" into a number that either goes up or goes down when someone changes the prompt. If the retrieval half of that pipeline is where you suspect the real problem lives, how to test a RAG pipeline is the broader end-to-end guide this metrics work sits inside.
Then stop, because the framework is finished and the temptation at this point is to add a fifth metric instead of the layer above. Four scores tell you the pipeline returned a grounded, relevant answer. They cannot tell you the citation rendered as a working link, the answer landed in the right field on the next screen, or the refusal path drew the fallback state instead of an empty div, because none of those are properties of the text. Autonoma is what we built for that layer, and it's deliberately not a fifth metric: it computes no score and grades no model output. Our Planner derives end-to-end cases from the application code around the RAG feature, the Executor drives them against the running app, and the assertion is whether the product ended up in the right state. Keep the four metrics as your pipeline gate. Add a behavioral gate above them for the failures a perfect score is structurally blind to.
Frequently Asked Questions
A RAG evaluation framework is the combination of metrics, a benchmark dataset, and pass/fail thresholds used to score a retrieval-augmented generation system and catch regressions before they ship. For an app builder, the practical version narrows to four metrics (faithfulness, answer relevancy, context precision, context recall) computed against a benchmark built from the product's own documents and real user questions, wired into CI rather than left as a one-off report.
Faithfulness (does the answer stay grounded in the retrieved chunks), answer relevancy (does the answer address the question asked), context precision (is the retrieved set free of irrelevant noise), and context recall (did retrieval include the chunk that actually contains the answer). Public metric catalogs list fifteen to twenty additional metrics, most of which are variants of these four or matter only for narrower use cases than a typical product feature.
Build a small set, thirty to fifty question-and-answer pairs is a reasonable start, sourced from real user questions in support tickets or chat logs rather than invented by the team. Write a ground-truth answer and record the supporting chunk IDs for each question, version the set alongside the RAG pipeline code, and re-score it on every change instead of relying on a public leaderboard, which reflects someone else's documents and never predicts performance on your own.
Both compute the same four core metrics using the same underlying claim-decomposition and entailment approach. Ragas scores at the dataset level, which fits a batch-first workflow with many rows scored at once. DeepEval scores per test case, which fits a pytest-style workflow asserting on one scenario at a time. The choice is about workflow shape, not accuracy, and switching later is a small change, not a rewrite.
Set a separate threshold per metric instead of one blended average, since an aggregate hides exactly the failure you need to see. Because the LLM judge scoring each metric is non-deterministic, run each benchmark question multiple times and gate on a pass rate, such as four-of-five, rather than a single verdict. A gate that fails on normal judge variance gets disabled by the team within a sprint, which defeats the entire point of having one.
