Production RAG Evaluation Checklist: Test Retrieval Quality, Groundedness, and LLM Answers

Overview

Retrieval-augmented generation (RAG) combines search with an LLM so an application can answer questions using a defined knowledge base. Evaluation must therefore cover two connected systems: the retriever that selects context and the generator that uses that context. A fluent answer can still be wrong if the relevant document was never retrieved, while excellent retrieval can still produce a poor result if the model ignores, misreads, or overstates the supplied evidence.

A useful RAG evaluation framework separates quality into four questions:

• Retrieval: Did the system find the right source material?
• Groundedness: Is the answer supported by the retrieved material?
• Answer quality: Did the response address the user’s question accurately and completely?
• Operational fitness: Is the system fast, reliable, observable, and affordable enough for its intended use?

Do not reduce the evaluation to a single score. Track component results separately so a regression has a plausible diagnosis. For example, a change to chunking may improve retrieval recall while making the final prompt too long. A model change may improve answer style while increasing unsupported claims. The goal is not merely to rank versions; it is to identify which part of the production AI application needs attention.

Before choosing instrumentation, document the intended behavior. Define which questions the system should answer, which sources are authoritative, when it should decline, and what a useful citation looks like. Your evaluation criteria should follow those decisions rather than generic notions of “good AI.”

What to track

1. Build a representative evaluation set

Create a versioned dataset of realistic queries and expected evidence. Include common questions, ambiguous wording, multi-step questions, questions with no answer in the corpus, and queries that could retrieve similar but incorrect documents. For each item, record the expected answer or key points, the relevant document identifiers, and any important restrictions such as “do not infer beyond the source.”

Start with anonymized production questions when available, supplemented by carefully written cases from domain experts. Keep a separate holdout set that is not used while tuning prompts, chunk sizes, filters, or reranking. Otherwise, improvements may reflect overfitting to familiar examples rather than better general behavior.

2. Measure retrieval quality

For each query, inspect whether the relevant source appears in the returned results and where it appears. Useful retrieval measurements include:

• Recall at k: whether at least one relevant item appears within the top k results.
• Precision at k: how much of the returned context is relevant.
• Rank position: whether the best evidence consistently appears early enough for the model to use it.
• Context coverage: whether the retrieved passages contain all the facts needed for the answer.
• Duplicate and noise rate: whether repeated or irrelevant chunks consume the context window.

Review failures by source type. A retrieval problem may come from poor parsing, stale indexing, unsuitable chunk boundaries, missing metadata, weak query rewriting, or an overly restrictive filter. A vector database comparison alone will not resolve a corpus or ingestion problem.

3. Test groundedness and answer quality

Groundedness testing asks whether each material claim in the answer can be supported by the provided context. Mark unsupported claims separately from incorrect claims: an answer can be factually plausible but still fail the application’s requirement to rely on its sources.

Evaluate answers against a small, explicit rubric. For example, score each response for factual accuracy, completeness, relevance, clarity, appropriate uncertainty, and citation correctness. Use a combination of automated checks and human review. An LLM-based evaluator can help triage large test sets, but it should be calibrated against human judgments and inspected for inconsistent reasoning. For high-impact workflows, retain human review for sampled answers and critical cases.

Citations deserve their own checks. Verify that every citation points to a retrieved source, that the source actually supports the nearby claim, and that the displayed title, page, section, or passage is understandable to the user. A citation that merely exists is not necessarily a useful citation.

4. Track refusal and safety behavior

Add cases where the answer is absent, the question is outside scope, the documents conflict, or retrieved text contains instructions aimed at manipulating the model. The expected behavior may be a clear limitation, a request for clarification, or a controlled escalation. These tests belong in the RAG testing suite, not only in a separate security review. For related defensive patterns, see prompt injection defense patterns for RAG and tool-using apps.

5. Monitor production behavior

Log enough structured data to investigate a request without unnecessarily retaining sensitive content. Depending on your requirements, track query identifiers, retrieval latency, generation latency, model and prompt versions, document identifiers, result ranks, token usage, error types, user feedback, and evaluation outcomes. Redact or protect user content according to your application’s data-handling rules. The broader observability approach is covered in observability for LLM apps.

Cadence and checkpoints

Run the full evaluation suite whenever you change the embedding model, chunking strategy, document parser, metadata filters, query transformation, reranker, prompt, model, citation format, or refusal logic. Treat these components as a chain: a change that appears local can alter downstream behavior.

Use three checkpoints:

1. Pre-merge: Run a small, fast smoke set covering core questions, no-answer cases, citations, and known regressions.
2. Release candidate: Run the complete labeled set, compare against the current production version, and manually inspect newly failed cases.
3. Post-release: Sample real interactions, review feedback, and compare operational metrics with the pre-release baseline.

Review the evaluation dashboard at least monthly for an actively changing application, or quarterly for a stable internal system. A recurring review should include new user questions, failed searches, citation complaints, empty-result rates, latency, cost, and changes in the document corpus. Keep a changelog that connects metric movement to code, data, configuration, or model changes.

How to interpret changes

Look for patterns across metrics rather than reacting to one percentage. A drop in retrieval recall with stable answer quality may be hidden by an easy test set or redundant documents. A stable retrieval score with lower groundedness can indicate that the prompt, context ordering, or generation model changed. Better answer ratings alongside higher latency or cost may be acceptable for one workflow and unacceptable for another.

Use failure buckets to make the next action clear:

• Relevant evidence missing: inspect ingestion, chunking, metadata, query formulation, and indexing.
• Evidence present but ignored: inspect context ordering, prompt instructions, context length, and model behavior.
• Answer supported but incomplete: retrieve more complementary passages or require coverage of each requested part.
• Answer unsupported: strengthen groundedness instructions, citation validation, refusal behavior, and post-generation checks.
• Correct answer but poor experience: improve formatting, directness, clarification flows, and citation presentation.
• Quality stable but operations degraded: investigate latency, retries, token growth, index performance, and model routing.

Compare like with like. Keep the evaluation dataset, retrieval depth, prompt version, model configuration, and scoring rubric attached to each run. If you change the rubric, preserve the old results and explain the break in the trend line. For architectural choices, the guide to choosing an LLM application framework can help separate framework concerns from evaluation concerns.

When to revisit

Revisit this checklist on a monthly or quarterly schedule, and immediately after any material change to the application or its knowledge base. A new document collection, changed access rules, altered business terminology, model migration, prompt revision, or increase in user volume can invalidate previous assumptions.

At each review, take five practical steps:

1. Export a fresh sample of representative queries and label the most important failures.
2. Run the versioned regression suite and compare retrieval, groundedness, answer, citation, latency, and cost results.
3. Inspect every severe regression and a sample of apparent improvements by hand.
4. Update the corpus, expected evidence, rubric, or safeguards when the product’s intended behavior has changed.
5. Record the decision, owner, affected version, and next review date.

Keep a small “must not regress” set for critical workflows and a growing set of newly observed failures. This turns RAG evaluation from a launch exercise into a production habit. If your system also uses agents or long-term memory, evaluate those retrieval paths separately; the guide to AI agent memory architectures provides useful distinctions. The durable objective is straightforward: know what the system retrieved, know what the answer claims, and maintain evidence that the two remain aligned as the application changes.