50–150 Golden Queries: RAG Evaluation Playbook for Engineers

Evaluate retrieval and generation separately, not as one blended score. Measure context relevance, faithfulness, and answer relevance, the three legs of what practitioners call the RAG Triad, and start with a small, human-labeled golden set before you let any LLM judge near your metrics. Calibrate that judge against your gold answers, report Wilson confidence intervals instead of bare percentages, and slice every result by query type. Skip any of these four moves and you're not doing RAG evaluation. You're guessing with extra steps.

TL;DR:

• Evaluate retrieval and generation separately to identify specific failure points, focusing on context relevance, faithfulness, and answer relevance.
• Use metrics like Recall@K, Precision@K, MRR, and nDCG for retrieval tuning, especially paying attention to multi-hop questions and content coverage.
• For generation, verify faithfulness by fact-checking claims against retrieved context and measure answer relevance by semantic similarity to the original query.
• Build a small, human-verified golden set first to calibrate automated judges and detect issues like chunking problems or metadata gaps before scaling.
• Incorporate continuous monitoring, version control, and stress testing, including adversarial and multi-turn evaluations, into your CI/CD pipeline for reliable, production-grade RAG systems.

Table of Contents

• Why RAG Evaluation Is Different From Plain LLM Evaluation
• Evaluating Retrieval: Metrics, Interpretation, And Debugging Steps
• Evaluating Generation And Grounding: Faithfulness, Answer Relevance, And Citation Checks
• Reference-Free Evaluation And LLM-As-Judge: RAGAS, ARES, And Judge Best Practices
• Building Evaluation Datasets: Golden Sets First, Synthetic QA Second
• Statistical Rigor: Confidence Intervals, Sample Sizes, And Reliable Claims
• Stress Testing And Adversarial Evaluation: Prompt Injection, Fault Injection, And Multi-Turn Checks
• Operationalizing Evaluation: CI/CD, Monitoring, Regression Sets, And Versioning
• A Practical Checklist For Evaluating A RAG System End To End
• What I Do Differently When I Build Production RAG Features
• Hanad Kubat: A Fixed-Price Path From Prototype To Production RAG
• Sources
• FAQ

Why RAG Evaluation Is Different From Plain LLM Evaluation

A retrieval-augmented generation system fails in more places than a standalone language model, because it has more moving parts to fail in. Chunking splits your documents. Embeddings turn chunks into vectors. A retriever pulls candidates. A reranker (if you have one) reorders them. A prompt assembler stitches context into the query. A reader model generates the final answer. Six stages, six separate ways for things to go wrong, and from the outside they all look identical: a bad answer.

That's the core problem. A user asks about your refund policy and gets a confident, wrong response. Was the right chunk missing from the index? Did the retriever rank it too low to make the cut? Did the reader ignore good context and hallucinate anyway? A single end-to-end quality score can't tell you. Research on evaluating retrieval-augmented generation systems argues explicitly that hybrid RAG pipelines need separate retrieval and generation metrics rather than one aggregate number, because you can't fix what you can't localize.

This is where the RAG Triad earns its place as the standard mental model:

• Context relevance asks whether the retrieved chunks actually relate to the question, independent of what the model does with them.
• Faithfulness (also called groundedness) asks whether the generated answer is actually supported by that retrieved context.
• Answer relevance asks whether the final answer actually addresses what the user asked, regardless of grounding.

Redis's engineering guidance on RAG evaluation frames these three as the minimum viable diagnostic set, and for good reason: each maps to a distinct stage. Low context relevance points at your retriever or your chunking strategy. High context relevance but low faithfulness points at your prompt or your reader model ignoring good evidence. High faithfulness but low answer relevance often means the model answered a slightly different question than the one asked. Once you can attribute a failure to a stage, fixing it stops being guesswork.

Evaluating Retrieval: Metrics, Interpretation, And Debugging Steps

Retrieval evaluation comes down to four metrics, and each one tells you something the others don't.

Recall@K measures the fraction of relevant documents that appear somewhere in your top K results. If there are three documents that genuinely answer the query and two show up in your top 5, your Recall@5 is 0.67. This is the metric that catches a broken index: if it's low even at K=20, your embedding model or chunking strategy is losing the right content entirely, not just ranking it poorly.

Precision@K measures the fraction of your top K results that are actually relevant. Precision@5 of 0.4 means 2 of your 5 retrieved chunks are noise. High recall with low precision usually means your retriever casts too wide a net, which raises your reranker's workload and can dilute the context your reader model sees.

Mean Reciprocal Rank (MRR) looks at where the first relevant result lands. If the first genuinely useful chunk shows up at position 3, that query's reciprocal rank is 1/3. Average this across your test set. MRR is the metric to watch when your reader model only pays attention to the first chunk or two, which happens more often than teams expect once context windows get crowded with marginal results.

Normalized Discounted Cumulative Gain (nDCG) accounts for graded relevance and rewards putting the best results near the top, not just any relevant result. It's the most information-dense of the four, and the right one to track once you've moved past binary relevant/not-relevant labels into a graded scale.

Here's how to turn these numbers into action:

1. Recall@K flat and low across K values (say, under 0.7 even at K=20): your corpus or embedding model has a coverage problem. Check for missing documents, bad chunk boundaries that split key sentences, or an embedding model mismatched to your domain vocabulary.
2. Recall@K high but MRR and nDCG low: your retriever finds the right content but buries it. Add or retune a reranker before touching anything else.
3. Precision@K consistently under 0.5: your retriever over-fetches. Tighten similarity thresholds or reduce K before the noise reaches your reader model.
4. Metrics good in aggregate but bad on multi-hop queries: this is a chunking problem in disguise. Single chunks rarely contain the full answer to a question that requires connecting two facts, so check whether your retrieval step supports multi-chunk assembly at all.

Pro Tip: Run your retrieval metrics on a held-out slice of queries you know require information split across two or more documents. Aggregate Recall@K often looks fine right up until you isolate multi-hop questions, where it frequently drops by half. That gap is one of the most reliable early-warning signs of a chunking strategy that's too aggressive.

A corpus health check worth running monthly: measure the percentage of chunks that never get retrieved across your entire query log. If a meaningful chunk of your index sits permanently unused, either your chunking created near-duplicate content competing for the same query space, or that content genuinely doesn't match anything users ask, which is a product question more than an engineering one.

Evaluating Generation And Grounding: Faithfulness, Answer Relevance, And Citation Checks

Faithfulness is the metric that separates a trustworthy RAG system from a confident hallucination generator, and it needs a concrete, repeatable procedure, not a vibe check.

The standard approach: extract every discrete factual claim from the generated answer, then check each claim against the retrieved context to see if it's actually supported. Redis's evaluation framework recommends exactly this per-claim verification, because a single hallucinated sentence buried in an otherwise-accurate paragraph will pass a holistic "does this look right" review far too often. Break the answer into atomic statements, mark each as supported, contradicted, or unsupported by the context, and calculate a faithfulness rate as supported statements divided by total statements.

Answer relevance works differently. It doesn't care whether the answer is grounded, only whether it addresses the question asked. A common approach: generate several questions that the answer would plausibly be responding to, then measure semantic similarity between those generated questions and the original query. If the reverse-engineered questions drift far from the actual query, the model likely answered something adjacent rather than the real thing.

Citation handling deserves its own scrutiny, because teams routinely conflate three separate properties:

• Citation presence: does the answer include citations at all, or just assert facts unattributed?
• Citation correctness: do the cited sources actually contain the claim they're attached to, or did the model cite something real but irrelevant?
• Citation sufficiency: do the citations, taken together, actually cover every claim made, or are some statements floating unsupported even in an answer that looks well-cited?

A system can score well on presence while failing badly on correctness, which is arguably worse than no citations at all, since it manufactures false confidence.

One more check that gets skipped constantly: refusal correctness. When a question has no good answer in the corpus, does the system say so, or does it fabricate one? Build a slice of your test set specifically from queries with no retrievable answer, and measure how often the system correctly declines instead of confabulating. A RAG system that never refuses anything is not accurate. It's just quiet about its failures.

Reference-Free Evaluation And LLM-As-Judge: RAGAS, ARES, And Judge Best Practices

Manual per-claim review doesn't scale past a few hundred examples, which is where reference-free, LLM-judged frameworks take over. Two of the most cited approaches, RAGAS and ARES, solve this with different trade-offs.

RAGAS uses an LLM directly as the judge for faithfulness, answer relevance, and context relevance, with prompts designed to align closely with human judgment without requiring reference answers. It's fast to stand up and works well once you trust the judge model, but every query costs an inference call to a capable LLM, which adds latency and expense at scale.

ARES takes a different path: it generates synthetic training data from your own corpus, fine-tunes a lightweight classifier to act as the judge, and then applies prediction-powered inference (PPI) to produce confidence intervals from a small number of human-labeled examples. The insight worth stealing here is that ARES gets strong accuracy with far fewer human annotations than a pure LLM-judge approach requires, because the PPI math statistically combines a small trusted labeled set with a much larger set of cheap model predictions.

Whichever framework you build on, judge reliability comes down to rubric design:

• Use a scale-based rubric, typically 1 to 5, with an explicit written definition for each point on the scale. A judge asked to just "rate faithfulness" drifts wildly between runs; a judge given "5 means every claim is directly and explicitly supported, 1 means the answer contradicts the context" does not.
• Ask the judge for a rationale before the score, not after. RAGAS's own findings show this ordering produces more consistent, defensible scores than a bare number.
• Force structured JSON output for every judge call. It's the difference between a metric you can aggregate reliably and a wall of free text you have to re-parse by hand.

Pro Tip: Never deploy an LLM judge without first running it against your golden set and checking agreement with your human labels. If the judge disagrees with your own reviewers more than 15 to 20 percent of the time on a set you already trust, the problem is the rubric, not your production system. Fix the judge before you fix the pipeline.

Building Evaluation Datasets: Golden Sets First, Synthetic QA Second

Every reliable RAG evaluation setup starts the same way: with a small dataset a human actually looked at, not a large one nobody has time to check.

1. Build the golden set manually first. Pull 50 to 150 real or representative queries, have a human write or verify the ideal answer and identify exactly which document chunks should be retrieved. Guidance from ML Digest's evaluation framework is blunt about why this order matters: skipping straight to automated judges without a human-verified baseline creates what it calls evaluation debt, where you're confidently measuring against a target that was never actually correct.
2. Use the golden set to calibrate any automated judge or metric before trusting it on new data. This is the step teams skip under deadline pressure, and it's the one that determines whether your dashboard numbers mean anything six months from now.
3. Generate synthetic QA pairs from your corpus to scale beyond what manual labeling can cover. Hugging Face's evaluation cookbook walks through generating question and answer pairs directly from document chunks, which gives you volume fast.
4. Run a critique pass on synthetic pairs before trusting them. Auto-generated questions frequently end up answerable without the source document, or too vague to have one correct answer. A filtering agent (or a second LLM pass) that flags and drops low-quality generated pairs keeps your expanded set from diluting your real signal.
5. Slice the full dataset deliberately, not just by volume. At minimum: single-hop questions answerable from one chunk, multi-hop questions requiring two or more chunks connected together, deliberately unanswerable questions to test refusal, freshness-sensitive questions where the correct answer depends on recency, and adversarial phrasings designed to trip up the retriever or the reader.

Sizing guidance that holds up in practice: aim for at least 20 examples per slice before drawing any conclusion about that slice, and treat 100 as the point where a metric starts feeling stable rather than noisy. Below 20, a single bad example can swing your reported rate by several points, which brings us to the statistics.

Statistical Rigor: Confidence Intervals, Sample Sizes, And Reliable Claims

A faithfulness rate of "92%" means almost nothing without knowing how many examples produced it. Report a confidence interval alongside every rate-based metric, and use the Wilson score interval, not the more common Wald interval, to compute it.

The reason is mechanical: the Wald interval (the simple "proportion plus or minus 1.96 times standard error" formula most people learn first) breaks down badly at small sample sizes and near the extremes of 0% or 100%, which describes most early-stage RAG evaluation runs exactly. Applied statistical guidance for RAG pipelines recommends Wilson specifically because it stays reliable in these small-sample, high-or-low-rate conditions where teams actually operate.

Sample size changes what you can honestly claim:

When comparing two pipeline versions, check whether their Wilson intervals overlap. If they do, you don't have a statistically distinguishable difference yet, regardless of which point estimate is higher. This single check prevents most of the false "we improved retrieval" claims that get walked back a month later.

Pro Tip: When your labeled sample is too small to trust on its own but too expensive to grow quickly, borrow the prediction-powered inference approach ARES uses: combine a small trusted human-labeled set with a larger set of model-predicted labels to get a tighter, still statistically valid interval, rather than choosing between "too small to trust" and "too expensive to label."

Stress Testing And Adversarial Evaluation: Prompt Injection, Fault Injection, And Multi-Turn Checks

Standard evaluation sets show you how a system performs on the queries you expected. Stress testing shows you what happens on the ones you didn't.

Prompt injection tests deliberately embed instructions inside retrieved content, things like a document chunk containing text that tells the model to ignore its system prompt or reveal internal instructions, then check whether the reader model follows the injected instruction or stays on task. Retrieval fault injection works on the other side of the pipeline: deliberately feed the system irrelevant or contradictory chunks and measure whether the reader notices and either flags the conflict or ignores the bad context, rather than blending it into a confused answer.

Concrete checks worth building into a stress-test suite:

• Inject a chunk with conflicting information alongside the correct one and measure whether the answer reflects the correct source or gets confused between them.
• Feed a query with no relevant chunks in the corpus at all and confirm the system refuses rather than fabricates.
• Test queries phrased as if the user already believes something false, and check whether the system corrects the premise or plays along.
• Run the same question with a slightly reworded phrasing and check for consistency in the answer's core claims.

Redis's engineering recommendations frame this as building containment gates: explicit checkpoints that stop a response before it reaches the user if it fails a faithfulness or relevance threshold, paired with a defined abstain policy for when the system genuinely doesn't know.

Multi-turn conversations need their own metric category, because hallucinations compound across a session in ways single-turn evaluation never surfaces. Track a session-level containment rate: the percentage of multi-turn conversations that stay fully grounded across every turn, not just the first one. A system that's 95% faithful per-turn can still produce a session where an early small inaccuracy gets built on by turn four into something the user fully believes and acts on.

Operationalizing Evaluation: CI/CD, Monitoring, Regression Sets, And Versioning

Evaluation that only runs manually before a big launch catches nothing between launches. Wire it into the same pipeline that ships code.

1. Gate deployments on metric thresholds in CI. Run your golden set and a fixed regression slice on every pull request that touches retrieval, prompts, or the reader model, and block merges that drop faithfulness or Recall@K below an agreed floor.
2. Monitor RAG Triad metrics continuously in production, not just at deploy time, sampled from real traffic rather than only the test set. Pair this with latency and containment rate (the percentage of low-confidence responses correctly caught before reaching the user) so a regression in quality doesn't hide behind healthy uptime numbers.
3. Maintain a fixed regression set that never changes unless you deliberately expand it, so metric drift over time is measuring your system, not a shifting target.
4. Version every component that can change behavior: the corpus snapshot, the vector index, the embedding model, the retriever configuration, the prompt template, and the evaluator itself. Redis's production guidance notes that metric drift more often follows a corpus update or a prompt tweak than a model upgrade, which means you need to know exactly what changed to explain why a number moved.
5. Store evaluation metadata alongside results: which corpus version, which prompt hash, which evaluator version produced a given score. Without this, a "why did faithfulness drop 4 points last Tuesday" investigation turns into archaeology.

For teams tracking these metrics over time without building custom dashboards from scratch, tools built for LLM analytics can handle the ongoing monitoring layer while your evaluation pipeline handles the correctness checks. For a broader operational view of how evaluation metrics fit into a stabilizing production system after a RAG feature ships, see these app stabilization strategies.

A Practical Checklist For Evaluating A RAG System End To End

If you're setting this up for the first time, the order matters as much as the individual steps.

1. Write 50 to 150 golden queries with human-verified answers and correct source chunks. Don't skip this to save a week. It's the baseline every later number gets compared against.
2. Compute retrieval metrics (Recall@K, Precision@K, MRR, nDCG) against the golden set and fix any obvious chunking or embedding issues before touching generation.
3. Manually score faithfulness and answer relevance on the same golden set using the per-claim extraction method, no automation yet.
4. Build an LLM-judge rubric with 1 to 5 scale definitions and rationale-first prompting, then run it against the golden set and check agreement with your manual scores.
5. Only trust the judge once agreement clears roughly 80 to 85 percent with your human labels. Below that, revise the rubric, not the pipeline.
6. Generate a larger synthetic QA set from your corpus, critique-filter it, and slice it into single-hop, multi-hop, unanswerable, freshness-sensitive, and adversarial categories.
7. Run stress tests, prompt injection and retrieval fault injection, and set containment gates before wide release.
8. Wire the golden set and regression slice into CI, gating merges on threshold drops.
9. Schedule a recurring regression run (weekly is a reasonable default for an actively developed system) against production traffic samples, and version every artifact involved.

Pro Tip: In the first two weeks, apply manual review almost everywhere and automation almost nowhere. It feels slow. It's the only way to know your automated judge is measuring the right thing once you do turn it loose at scale.

For teams that want the automated-metrics layer built out further once the manual baseline is solid, this breakdown of AI evaluation metrics for production LLMs covers how to localize whether a regression traces back to retrieval or generation.

What I Do Differently When I Build Production RAG Features

Most teams I see building RAG features skip straight to an automated eval framework because it feels like progress. I don't. I keep the golden set small and human-reviewed first, usually under 100 examples, because a judge calibrated against a bad baseline just launders bad answers with a confident-looking score attached.

The recurring problem I fix during rebuilds isn't the model. It's the source of truth underneath it: chunking that splits a policy document mid-sentence, metadata that's missing so the retriever can't filter by date or product line, and a reader model with no instruction to refuse when the context genuinely doesn't answer the question. Fix those three and most "hallucination" complaints disappear before you touch a single prompt.

This is also where a fixed-price, fixed-scope build actually helps rather than constrains. When the evaluation loop gets built early, in week one alongside the retriever, not bolted on after launch, you catch the chunking and metadata problems while they're a day's work instead of a production incident. Scope frozen at kickoff means that loop is part of the spec, not a favor I'm asking for later.

— Hanad Kubat

Hanad Kubat: A Fixed-Price Path From Prototype To Production RAG

If your RAG feature works in a demo but you're not sure it holds up past 100 real queries, that gap is exactly what Hanad Kubat closes for B2B SaaS teams: production-grade evaluation and monitoring built into the same fixed-price engagement as the feature itself, not a separate line item you get to later. There's no ongoing retainer forcing the scope, no offshore markup, and no project-manager layer between you and the person writing the code. Every line is written by me, no juniors, and you own the code from the first commit.

This fits teams shipping their first real RAG feature and teams rescuing one built fast in a prototyping tool that now needs to survive due diligence, a security review, or actual production traffic. If EU AI Act or GDPR obligations apply to your use case, that gets built into the architecture from the start rather than patched in afterward; see this practical checklist for EU AI Act compliance for what that involves.

Start with the fixed-price Prototype Audit, three to five days, credited against the eventual build, to get a concrete assessment of where your current retrieval and generation pipeline actually stands before committing to a full rebuild.

Sources

A few sources worth reading in full if you're implementing this rather than just skimming it:

• Evaluation of Retrieval-Augmented Generation: A Survey
• RAG system evaluation (Redis blog)

FAQ

Is ChatGPT A RAG Model?

No. ChatGPT is a large language model that generates answers from what it learned during training. It becomes a RAG system only when connected to an external retrieval step, like its browsing or file-search features, that pulls in outside documents at query time.

How Do I Test My RAG System?

Build a small human-labeled golden set of 50 to 150 queries with verified answers and correct source chunks, then measure retrieval metrics (Recall@K, Precision@K, MRR, nDCG) and generation metrics (faithfulness, answer relevance) against it before scaling to automated LLM judges.

What Are Some Evaluation Tools For RAG Methods?

RAGAS and ARES are the two most widely referenced automated frameworks, offering LLM-judge and fine-tuned-judge approaches respectively. For ongoing production monitoring rather than one-off testing, dedicated LLM analytics tooling and multi-LLM audit tools also fill part of the gap.

What Is RAG vs LLM?

An LLM generates answers purely from its trained parameters, with no access to your specific documents. A RAG system adds a retrieval step that pulls relevant documents at query time and feeds them to the LLM as context, which is why RAG evaluation has to score retrieval and generation as separate stages instead of one blended output.

Recommended

• Engineering Best Practices List for Software Teams in 2026
• Engineering Standards for Startups: A Practical Playbook
• Reliable Engineering for SaaS CTOs: A Practical Guide
• Stop Regressions: AI Evaluation Metrics for Production LLMs