How to Evaluate a RAG Pipeline: Retrieval, Accuracy, Hallucinations and Production Evals

To evaluate a RAG pipeline means to test two systems, not one. First, you need to determine whether the retrieval layer found the evidence required to answer the question. Then you need to determine whether the language model used that evidence correctly.

That distinction matters because a convincing answer can hide poor retrieval, while good retrieval can still produce an inaccurate or hallucinated response.

A useful RAG evaluation framework therefore combines retrieval metrics such as Recall@k, Precision@k and rank-sensitive measures with generation metrics including faithfulness, correctness and relevance. It should also test unanswerable questions, latency, context usage and production regressions.

Most importantly, every configuration should be evaluated against the same versioned question set so that improvements are measurable rather than subjective.

RAG Has Two Separate Failure Surfaces

A typical retrieval-augmented generation pipeline looks simple:

Question → Retrieval → Retrieved context → Generation → Answer

But that pipeline creates at least two distinct places where quality can break down. The retrieval system can fail to find the correct document or passage.

Or the retrieval system can find exactly the right evidence, only for the model to misinterpret it, ignore part of it or introduce a claim that the evidence does not support.

That distinction should drive the entire RAG evaluation process.

Consider a company knowledge assistant asked: What is the cancellation period in our current enterprise agreement?

If the relevant contract clause never appears in the retrieved context, the problem is primarily retrieval. If the correct clause is retrieved but the model answers “30 days” when the document says “60 days,” retrieval worked and generation failed.

If the retrieved context contains several obsolete agreements and the current contract is buried at rank eight, the problem may be ranking.

And if the corpus contains no cancellation clause at all but the assistant confidently invents one, the system has failed its abstention and grounding controls.

An end-to-end “answer quality” score can hide all four cases. This is why RAG evaluation should begin with failure attribution rather than a long list of metrics.

The evaluation pipeline

A useful evaluation architecture separates the layers explicitly:

Layer	Question
Input	Is this a realistic and correctly labelled test question?
Retrieval	Did the system find the evidence needed to answer it?
Ranking	Did useful evidence appear early enough?
Context	Did we send too much or too little information to the model?
Generation	Did the model answer correctly using the supplied evidence?
Grounding	Are the answer’s claims supported by that evidence?
Control	Did the system abstain when no supported answer existed?
Operations	What did the query cost in latency and context usage?

Once those layers are visible, the metrics become much more useful.

The RAG Metrics That Actually Matter

There is no single “best RAG metric.” A production system normally needs a small combination of metrics because each one detects a different class of failure.

Recall@k: Did retrieval find the evidence?

Recall@k asks whether the relevant evidence appears somewhere in the first k retrieved results. Suppose a question has two known relevant passages. If both appear in the top five results, recall is high. If only one appears, recall is lower.

Recall is especially important when missing evidence is more damaging than retrieving some irrelevant evidence.

For example, an internal compliance assistant may need every relevant clause required to answer a question. Retrieving one correct clause while omitting another material exception can still produce a misleading answer.

A low Recall@k score usually points toward retrieval problems such as:

• poor embeddings for the domain;
• unsuitable chunk boundaries;
• incomplete indexing;
• weak query transformation;
• metadata filtering errors; or
• a value of k that is too restrictive.

High recall does not mean the retrieved context is good overall. It only means the evidence was found.

Precision@k: How much retrieved context is actually useful?

Precision@k measures how much of the retrieved material is relevant. If the system retrieves five chunks and only one helps answer the question, it may technically have found the right evidence, but it has also surrounded that evidence with noise.

That matters because RAG systems do not operate in an unlimited context environment.

Irrelevant chunks consume tokens, increase processing time and can distract the generator from better evidence. A system can therefore have strong recall and poor precision at the same time.

For example:

• Configuration A retrieves the correct passage plus nine irrelevant chunks.
• Configuration B retrieves the same passage plus two closely related chunks.

Both systems found the answer. Configuration B gives the language model a cleaner context.

Some RAG evaluation frameworks distinguish ordinary retrieval precision from context-precision measures that also consider ranking. Ragas, for example, defines context precision around whether relevant chunks appear above irrelevant ones in the retrieved context.

The key point is not the library-specific implementation. It is that you should measure whether the retriever is sending useful evidence rather than merely retrieving more text.

MRR: How quickly does the first useful result appear?

Mean Reciprocal Rank, or MRR, is useful when the rank of the first relevant result matters.

If the best supporting passage is consistently returned first, the system is behaving very differently from one where the same passage regularly appears at rank eight or nine. MRR rewards systems that place a relevant result near the top.

For a single query:

Reciprocal rank = 1 / rank of the first relevant result

A first-place relevant result receives 1.0. A second-place result receives 0.5. A fifth-place result receives 0.2. MRR then averages these reciprocal ranks across the evaluation set.

For more complex retrieval tasks where several relevant documents have different degrees of usefulness, metrics such as nDCG can be more informative.

You do not need every ranking metric in every RAG project. Use them when ranking behaviour materially affects what reaches the model.

Faithfulness: Is the answer supported by the retrieved context?

Retrieval metrics tell you what evidence was found. Faithfulness tells you what the model did with it. A faithful answer should make claims that can be supported by the context supplied to the generator.

That makes faithfulness one of the most useful signals for detecting RAG hallucinations. An answer can be factually true but still be unfaithful to the retrieved evidence.

Suppose a model knows from pretraining that a company was founded in 2011, but the retrieved company documents do not state a founding date. If the model nevertheless supplies “2011,” the statement may happen to be correct, but the RAG answer is not grounded in the provided context.

That distinction is critical in enterprise systems where provenance matters.

Correctness: Did the system produce the right answer?

Faithfulness and correctness are related but not identical.

Faithfulness asks: Does the evidence support what the model said?

Correctness asks: Is the answer actually right according to our reference answer or acceptance criteria?

A model can faithfully summarize the wrong document. If retrieval supplies an outdated policy and the model accurately describes that outdated policy, generation may be faithful while the final answer remains incorrect.

That is one reason a golden dataset should include expected documents or passages as well as reference answers.

Relevance: Did the answer address the question?

A response can be grounded and factually correct while still doing a poor job of answering the user’s actual question. Answer relevance catches this.

For example, a user asks: Can contractors access the system from personal devices?

The assistant responds with a correct four-paragraph description of the company’s general security policy but never states whether personal devices are permitted. The response may contain no hallucinations at all, yet it has failed the user’s task.

Abstention: Does the system know when it cannot answer?

Abstention is one of the most overlooked RAG evaluation metrics. Every golden evaluation dataset should contain questions the corpus cannot answer.

Those questions test whether the system says, in effect: I don’t have sufficient evidence in the available documents to answer that.

An assistant that produces polished answers to impossible questions may score well on perceived fluency while performing dangerously in production.

For private knowledge assistants, legal-document systems, policy tools and regulated applications, correct refusal can be as important as correct answering.

Latency and context-token usage

Quality metrics should not be evaluated in isolation. Increasing k from five to 25 might improve recall. It can also increase context size, reranking work, inference time and cost.

Likewise, a sophisticated reranker might improve retrieval precision while adding unacceptable latency.

For every RAG experiment, record at least:

• retrieval latency;
• end-to-end response latency;
• number of retrieved chunks;
• context size or tokens;
• answer length where relevant; and
• failure/error rate.

A change that improves a quality metric by two percentage points while doubling latency is not automatically an improvement.

Build a Golden RAG Evaluation Dataset

The most valuable part of a RAG evaluation system is usually not the evaluation framework. It is the dataset. A golden dataset gives you a fixed, versioned collection of questions against which different retrieval and generation configurations can be compared.

Without a fixed set, evaluation quickly becomes anecdotal. You change chunk size, ask three questions manually, notice the answers look better and conclude the new configuration works.

That is not a reliable evaluation.

What each golden example should contain

At minimum, an answerable RAG evaluation case should contain:

Field	Example
Question	What is the notice period for terminating the agreement?
Expected document	Enterprise Services Agreement v3
Expected passage	Section 14.2
Reference answer	60 days’ written notice
Acceptance criteria	Must state 60 days and identify written notice requirement
Category	Contract / direct retrieval
Answerable	Yes

For an unanswerable question:

Field	Example
Question	What penalty applies for cancelling after 10 PM?
Expected document	None
Expected passage	None
Reference answer	Insufficient information
Acceptance criteria	Must not invent a penalty
Category	Abstention
Answerable	No

The relevant passage label is particularly useful because it lets you separate retrieval failure from generation failure. If the expected passage was never retrieved, the generator was not given a fair opportunity to answer. If it was retrieved and the answer was still wrong, you know to investigate the generation layer.

How large should a RAG evaluation set be?

There is no universal minimum that turns a small evaluation set into a production benchmark. Start with enough high-quality examples to represent the important behaviours of the system, then grow the dataset using real failures.

For an initial engineering evaluation, dozens of carefully reviewed questions can be more useful than hundreds of poorly labelled synthetic ones.

The AIMEC evaluation described in this article uses a deliberately bounded test set rather than claiming to benchmark every RAG workload.

As the system reaches production, the set should expand to include:

• common user questions;
• edge cases;
• ambiguous questions;
• multi-document questions;
• terminology variations;
• difficult retrieval cases;
• stale/versioned document cases;
• adversarial or misleading prompts;
• unanswerable questions; and
• failures discovered from real production traffic.

The important property is not merely dataset size. It is coverage.

Version your golden dataset

Treat the evaluation set as a software artifact. If the questions, expected passages or reference answers change between experiments without being tracked, your before/after comparison becomes unreliable.

A useful version might record:

• dataset version;
• corpus version;
• retrieval configuration;
• embedding model;
• generator model;
• prompt version;
• reranking settings;
• evaluation model or judge;
• run date; and
• code commit.

This turns RAG evaluation into a reproducible engineering process instead of an occasional manual test.

How AIMEC Evaluated Its Retrieval Pipeline

For this evaluation, AIMEC used the Local Agent Harness as a bounded RAG test environment. The objective was not to establish a universal benchmark for retrieval frameworks or language models. It was to answer three narrower engineering questions:

1. Did the retrieval system reliably find the evidence required to answer a fixed set of questions?
2. Did reranking or wider retrieval improve the evidence supplied to the generator?
3. What happened to answer coverage, abstention, context size and latency when the retrieval configuration changed?

Test environment

The evaluation used a controlled corpus of eight enterprise-style Markdown documents covering security policy, data retention, deployment architecture, incident response, model runtime, private-network governance, procurement and support operations.

A fixed golden set contained 40 questions:

• 15 straightforward factual questions;
• eight semantic or paraphrased questions;
• five cross-section questions;
• four multi-document questions;
• three difficult or ambiguous questions; and
• five deliberately unanswerable questions.

That produced 35 answerable cases and five abstention cases.

Every answerable question was labelled with the document and section expected to contain the required evidence, together with deterministic answer acceptance criteria. The same 40 questions were used for every configuration.

Each configuration therefore generated 40 runs, for a total of 120 RAG executions.

The test used the Local Agent Harness’s existing section-aware indexing pipeline rather than introducing a benchmark-specific chunking method. Documents were indexed as titled paragraphs, with oversized paragraphs split when necessary.

The retrieval and generation stack was:

• Embedding model: BGE small English v1.5 Q8_0;
• Embedding dimensions: 384;
• Vector store: local Qdrant;
• Foreground model: approximately 7.62 billion parameters;
• Model format: GGUF;
• Quantization: Q5_K Medium;
• Runtime: llama.cpp;
• Loaded foreground context: 8,192 tokens;
• Reranker: cross-encoder/ms-marco-MiniLM-L6-v2;
• Evaluation dataset version: aimec-rag-eval-v1;
• Evaluation date: 6 September 2026.

Three retrieval configurations were tested.

Configuration A: normal retrieval with reranking

The normal retrieval configuration used a similarity threshold of 0.45 and considered up to 12 vector-search candidates. A cross-encoder reranker then selected the strongest excerpts for generation.

Configuration B: normal retrieval without reranking

The second configuration used the same similarity threshold and candidate limit but removed the cross-encoder reranking stage.

This provided a controlled way to test whether reranking improved the final evidence context or whether it discarded useful information.

Configuration C: deep retrieval with reranking

The deep configuration lowered the similarity threshold to 0.30 and expanded the candidate pool to 24 before reranking.

This tested a common RAG assumption: that searching more broadly and supplying more candidate evidence will improve answer quality.

Retrieval was scored independently using Recall@k, Precision@k and MRR. The benchmark also recorded a deterministic answer-acceptance score, abstention behaviour, end-to-end latency and retrieved-context usage.

The deterministic answer score checked whether required facts appeared in the final response. It should not be interpreted as a complete measure of semantic correctness or faithfulness.

What Changed When AIMEC Changed Retrieval and Chunking?

In this experiment, AIMEC held its section-aware chunking approach constant. The actual variables were retrieval depth and whether the retrieved candidates were passed through the cross-encoder reranker.

The results showed why RAG changes should be evaluated across several metrics rather than judged by a single accuracy score.

Metric	Baseline + reranker	No reranker	Deep + reranker
Recall@k	0.929	1.000	0.929
Precision@k	0.293	0.219	0.195
MRR	0.971	0.950	0.971
Deterministic answer acceptance	0.879	0.990	0.879
Abstention accuracy	0.600	0.400	0.400
Median end-to-end latency	5.68 s	6.47 s	10.93 s
Mean retrieved context	138 estimated tokens	416 estimated tokens	203 estimated tokens

No configuration won every metric.

Reranking produced much cleaner context

The largest operational difference appeared in context usage.

Without reranking, the system supplied approximately 416 estimated context tokens per question on average. With the normal reranking configuration, that fell to approximately 138.

That is a reduction of roughly 67%.

Precision@k also improved from 0.219 without reranking to 0.293 with reranking, while MRR increased from 0.950 to 0.971.

This indicates that the reranker was generally successful at removing weaker candidates and moving useful evidence toward the top of the context.

Median latency also fell from 6.47 seconds to 5.68 seconds.

For this workload, reranking therefore did not simply add another expensive model stage. The smaller final context appears to have offset at least part of the reranking cost.

The trade-off: reranking also removed useful evidence

The same configuration exposed an important failure mode.

Recall@k fell from 1.000 without reranking to 0.929 after reranking.

The underlying vector retrieval stage was therefore able to find all expected evidence across the answerable test cases, but the final reranking and selection process did not preserve all of it.

That distinction matters.

A reranker can improve precision and ranking quality while still damaging task performance if it filters out a secondary passage needed to answer a multi-section or multi-document question.

The answer-acceptance results moved in the same direction.

The configuration without reranking achieved a deterministic acceptance score of 0.990, compared with 0.879 for the reranked configurations.

This suggests that the larger context sometimes preserved facts the generator needed even though that context was less precise overall.

It would therefore be misleading to say simply that reranking “improved retrieval.” It improved some retrieval properties—particularly precision, ranking and context efficiency—but reduced evidence coverage on this test set.

Wider retrieval was not better

The deep retrieval configuration produced the clearest negative result.

Lowering the similarity threshold from 0.45 to 0.30 and doubling the candidate pool from 12 to 24 did not improve Recall@k. Recall remained at 0.929.

Precision fell from 0.293 to 0.195.

Mean retrieved context increased from approximately 138 to 203 estimated tokens.

Most significantly, median end-to-end latency increased from 5.68 seconds to 10.93 seconds—roughly a 93% increase.

The wider search therefore increased processing and context without finding additional required evidence.

On this controlled corpus, “retrieve more” was not an effective optimization strategy.

Abstention remained a weakness

The five unanswerable questions produced another important finding.

The baseline reranked configuration correctly abstained on three of the five unsupported questions, producing an abstention accuracy of 60%.

Both the non-reranked and deep configurations correctly handled only two of five, or 40%.

Retrieval quality was therefore considerably stronger than unsupported-query handling.

This is important because good Recall@k and MRR only describe behaviour when relevant evidence exists. They do not guarantee that a generator will refuse to answer when the corpus contains no evidence at all.

For high-risk RAG applications, abstention requires its own evaluation and potentially its own control layer.

What the experiment actually showed

The strongest production configuration was not the configuration with the highest value for every metric.

Normal retrieval with reranking produced:

• substantially less context;
• better retrieval precision;
• stronger ranking;
• lower median latency;
• and better abstention behaviour.

But removing reranking produced:

• perfect retrieval recall;
• and the highest deterministic answer-acceptance score.

Meanwhile, deep retrieval added latency and context without increasing recall.

The practical conclusion is that retrieval configuration should be treated as a multi-objective optimization problem. Recall, precision, ranking, answer behaviour, abstention and operational cost can move in different directions.

Diagnosing Common RAG Failures

The AIMEC experiment produced several failure patterns that illustrate why metrics are most useful when they lead to an engineering diagnosis.

Symptom	Evidence from the AIMEC test	Likely cause	Metric to inspect	Possible fix
Required evidence disappears after candidate selection	Recall fell from 1.000 without reranking to 0.929 with reranking	Reranker or final context selection is too aggressive	Recall@k before and after reranking	Increase final result limit, adjust reranking policy or preserve evidence diversity
Correct evidence is surrounded by more irrelevant material	Precision fell from 0.293 with reranking to 0.219 without it	Too many raw vector candidates reach generation	Precision@k, context size	Rerank or filter before generation
Wider search retrieves more text but no more required evidence	Deep retrieval kept recall at 0.929 while precision fell to 0.195	Similarity threshold is too permissive	Recall@k + Precision@k	Tighten threshold or reduce candidate pool
Retrieval change dramatically increases runtime	Deep median latency rose from 5.68 s to 10.93 s	Wider retrieval and reranking workload	Latency	Reduce candidate pool or reserve deep mode for questions that need it
Context expands without corresponding retrieval benefit	Context grew from 138 to 203 tokens in deep mode without recall improvement	Excess retrieval	Context usage + Recall@k	Tighten retrieval before generation
Removing context filtering improves answer acceptance	No-reranker acceptance reached 0.990 versus 0.879 with reranking	Useful secondary evidence may have been pruned	Recall + answer acceptance	Preserve multiple relevant sections or tune reranker limits
System answers questions with no supporting evidence	Best abstention accuracy was only 60%	Weak answerability or grounding control	Abstention test set	Add explicit evidence gating, refusal rules or answerability scoring

This experiment also demonstrates why a poor answer should not automatically trigger a generator upgrade.

If the required evidence was available after vector search but removed by reranking, changing the language model would not solve the actual failure.

Likewise, increasing retrieval depth would not solve the deep-mode result observed here. The system already found the same amount of required evidence; the wider search merely added noise and latency.

The correct engineering action depends on the layer that failed.

Human Evaluation vs LLM-as-a-Judge

Automated evaluation is useful because manually reviewing every RAG answer quickly becomes impractical. But the AIMEC benchmark also illustrates why different types of automated scoring should not be confused.

The reported answer-acceptance metric was deterministic. Each answerable golden example contained required facts, and the evaluator checked whether those required terms appeared in the generated response.

This produced scores of:

• 0.879 for baseline retrieval with reranking;
• 0.990 without reranking; and
• 0.879 for deep retrieval with reranking.

Those numbers are useful for regression testing because the same rule can be applied consistently after every system change.

They are not equivalent to a full correctness score.

A response can contain the expected phrase while still misrepresenting its meaning. It can omit an important qualification. It can include the correct answer alongside an unsupported claim. Conversely, a semantically correct paraphrase may fail a poorly designed literal check.

The same caution applies to LLM judges.

LLM-based evaluators can scale subjective measures such as:

• answer relevance;
• faithfulness;
• semantic correctness;
• document relevance; and
• application-specific criteria.

But an LLM judge should not be treated as objective ground truth.

For the AIMEC evaluation workflow, deterministic checks and retrieval labels should provide the reproducible foundation. LLM-based evaluation can then add semantic coverage, while human review should adjudicate failures and borderline cases.

A practical process is:

1. run deterministic and retrieval metrics over the complete golden set;
2. run an LLM judge for properties such as faithfulness and relevance;
3. manually inspect failed and borderline answers;
4. sample apparently successful answers as well;
5. record cases where human judgement and automated evaluation differ; and
6. convert important disagreements into better acceptance criteria or deterministic regression tests.

The current controlled run establishes the retrieval and deterministic baseline. A separate judged review is still required before AIMEC should report LLM-judge or human-agreement statistics as measured results.

For high-risk applications, subject-matter experts may still need to define what constitutes an acceptable answer. Automated evaluation can prioritize review; it should not silently replace it.

Turn RAG Evals Into Regression Tests

The AIMEC test becomes substantially more useful if the same dataset is retained and run again after retrieval, model, prompt or indexing changes.

The first run has now created a reproducible baseline:

Fixed corpus → 40-question golden set → 3 configurations → 120 executions → recorded metrics

Future versions of the Local Agent Harness can be tested against the same dataset rather than compared through manual impressions.

For the next change, the current results can serve as reference values:

Regression signal	Current reference
Recall@k, standard reranked retrieval	0.929
Precision@k	0.293
MRR	0.971
Deterministic answer acceptance	0.879
Abstention accuracy	0.600
Median end-to-end latency	5.68 s
Mean retrieved context	138 estimated tokens

These should initially be treated as reference baselines rather than universal production thresholds.

For example, a future reranker could improve recall from 0.929 to 0.97 while increasing context modestly. That may be worthwhile.

A different change might increase precision while lowering recall further. Whether that is acceptable depends on the application.

The important requirement is that the trade-off is explicit before release.

A useful regression process is:

Baseline → Change → Run fixed eval set → Compare metrics → Inspect failures → Apply release gate → Deploy

Each important release should preserve:

• dataset and corpus versions;
• previous and current configuration;
• retrieval metrics;
• answer-evaluation metrics;
• failed question IDs;
• latency;
• context usage; and
• human-review notes.

This prevents a retrieval improvement in one area from silently degrading another.

At a more experimental level, the same test-measure-iterate pattern appears in AutoResearch, where AI is used to propose, run and evaluate iterative changes rather than relying purely on manual experimentation.

Production RAG Evaluation

The controlled AIMEC test also demonstrates why production RAG evaluation cannot focus only on answer accuracy.

Consider the deep retrieval configuration.

Recall remained unchanged at 0.929, yet median latency increased from 5.68 seconds to 10.93 seconds.

A quality-only evaluation could easily miss that regression because the system was still finding roughly the same required evidence.

The same applies to context usage.

Removing the reranker increased mean retrieved context from approximately 138 to 416 estimated tokens—roughly three times as much context.

That configuration did improve recall and deterministic answer acceptance, so the additional context cannot simply be labelled waste. But it demonstrates the type of trade-off that production monitoring should expose.

Useful production signals therefore include:

• no-result retrievals;
• low retrieval scores;
• unusually large contexts;
• changes in average candidate count;
• reranker failures;
• answers with missing evidence;
• unsupported answers;
• abstentions;
• user corrections;
• repeated reformulations;
• latency changes; and
• queries outside the golden dataset’s coverage.

The AIMEC results suggest that retrieval depth deserves particular attention.

If production queries begin triggering wider retrieval more frequently, latency could rise materially without a corresponding improvement in evidence coverage.

Similarly, if a reranker update makes contexts smaller, teams should verify that improved efficiency has not reduced recall.

A production failure should not merely be fixed once.

When appropriate, it should become a new golden case:

Production failure → Reviewed example → Golden dataset → Fix → Regression test

Over time, this process causes the evaluation dataset to represent the system people actually use rather than only the workload engineers anticipated.

This failure attribution becomes even more important when retrieval is one component inside a multi-agent application. Systems built with multi-agent development frameworks such as LangGraph or CrewAI can introduce additional orchestration, tool-use and agent-level failure surfaces around the RAG pipeline.

Watch for retrieval drift

The benchmark used a deliberately small, fixed corpus. Production corpora do not remain fixed.

Documents are added, replaced and duplicated. Policies change. Similar content accumulates. Internal terminology evolves.

A similarity threshold that works well across eight controlled documents may behave differently across thousands of enterprise files.

Corpus changes should therefore trigger evaluation just as code changes do.

Protect evaluation data

The AIMEC experiment used synthetic enterprise-style documents, which simplified evaluation privacy.

A production evaluation system will not have that luxury.

Prompts and retrieved context may contain employee records, confidential contracts, customer information or internal strategy.

Evaluation therefore requires its own privacy controls, including:

• access restrictions;
• tenant isolation;
• redaction where appropriate;
• retention rules;
• encryption;
• local or private evaluators; and
• controls over which evidence can be sent to external judge models.

A private RAG implementation loses much of its value if the evaluation pipeline becomes the route through which protected context leaves the environment.

A Practical RAG Release Gate

The AIMEC benchmark provides a useful example of why a release gate needs several dimensions.

If the decision had been based only on Recall@k, the no-reranker configuration would have won because it achieved 1.000.

If the decision had been based on precision, ranking, context efficiency and latency, the normal reranked configuration would have won.

If the decision had been based purely on deterministic answer acceptance, the no-reranker configuration would again have won.

The release decision therefore depends on which trade-offs are acceptable for the application.

For future AIMEC regression runs, the following framework can use the first experiment as its reference point:

Gate	Current AIMEC reference	Release interpretation
Dataset integrity	Same versioned 40-question set and eight-document corpus	Required for valid comparison
Retrieval recall	0.929 standard baseline	Any reduction requires investigation or an explicit trade-off
Retrieval precision	0.293 standard baseline	Lower precision should produce a measurable benefit elsewhere
Ranking	MRR 0.971	Relevant evidence should remain near the top
Answer acceptance	0.879 standard baseline	Must not regress without explanation
Abstention	0.600	Current weakness; future changes should target improvement rather than regression
Latency	5.68 s median	Large increases require corresponding quality gains
Context usage	138 estimated tokens	Growth should be justified by better evidence coverage or answer quality
Critical cases	No silent regression	Mandatory questions should be reviewed individually
Human review	Judged/human sample still required	Complete before high-risk production use
Reproducibility	Dataset, model and settings recorded	Required
Privacy	Evaluation remains local/private where required	Required

This table should not be interpreted as saying that 0.929 recall or 5.68-second latency is universally acceptable.

The first benchmark establishes a reproducible reference, not a production SLA.

The important discipline is to decide what a future version is allowed to degrade before looking at its results.

Otherwise, it is easy to justify whichever configuration happens to perform best on the metric a team prefers after the test is complete.

The Decision: Evaluate RAG by Failure Layer, Not by One Aggregate Score

A RAG system should not be considered production-ready because its answers sound good.

It should be possible to demonstrate where evidence came from, whether the retriever reliably finds that evidence, whether the model uses it correctly, how the system behaves when evidence is absent and whether those behaviours survive future changes.

That requires more than a metric dashboard.

It requires a versioned evaluation dataset, separate retrieval and generation tests, controlled before/after experiments, explicit failure analysis and a release gate that includes operational trade-offs.

The most useful question is therefore not:

“What is our RAG score?”

It is:

“When this system fails, can we identify which layer failed, reproduce the failure and prevent it from returning?”

That is the point at which RAG evaluation becomes an engineering discipline rather than a demo metric.

Frequently Asked Questions About RAG Evaluation

What is the best RAG metric?

There is no single best RAG metric because retrieval and generation fail differently. For retrieval, Recall@k is useful for determining whether required evidence was found, while precision and ranking metrics help determine whether the returned context is clean and well ordered. For generation, correctness and faithfulness answer different questions: whether the response is right and whether it is supported by the retrieved evidence. A production system should normally use a small metric set aligned with its actual failure risks.

What is RAG faithfulness?

Faithfulness measures whether claims in the generated response are supported by the context supplied to the model. It is therefore useful for identifying hallucinations or unsupported additions. Faithfulness should not be confused with correctness. A model can faithfully summarize incorrect or outdated retrieved information.

How large should a RAG evaluation set be?

There is no universal number that guarantees a representative evaluation. For an early controlled experiment, dozens of carefully curated examples can provide useful engineering feedback. Production systems should expand their datasets as new failure modes and real-world question types emerge. Coverage and label quality matter more than hitting an arbitrary question count.

Can an LLM judge RAG answers?

Yes, and LLM judges are useful for scaling subjective evaluations such as relevance, faithfulness and some forms of correctness. They should not be considered objective ground truth. Teams should manually review samples, investigate disagreements and use deterministic tests or subject-matter experts where the risk warrants it.