Production RAG Tutorial: Build, Evaluate, and Maintain a Retrieval-Augmented AI App

Overview

Retrieval-augmented generation, or RAG, combines a language model with a searchable collection of external knowledge. Instead of asking a model to answer entirely from its training data, the application retrieves relevant passages and places them in the model's context before generating a response. This pattern is useful for internal documentation, support content, product knowledge, research collections, and other domains where answers should reflect a controlled source of information.

A basic RAG pipeline has five stages:

1. Ingestion: collect documents and extract usable text and metadata.
2. Chunking: divide documents into passages that can be retrieved without losing important context.
3. Embedding and indexing: represent passages numerically and store them in a vector search system.
4. Retrieval: find candidate passages for a user's question, often with filtering or reranking.
5. Generation: provide the selected context to a language model and return an answer with appropriate citations.

The difficult work begins after the first successful response. A production-ready AI app must make failures visible. A response can be fluent but unsupported, technically relevant but incomplete, or based on an outdated document. Monitoring therefore needs to cover the complete path from source document to final answer rather than focusing only on model latency.

Keep the architecture understandable. A framework can accelerate development, but the team should still be able to inspect the retrieved chunks, prompts, metadata, model inputs, outputs, and evaluation results. For a broader discussion of framework choices, see how to choose a framework for building LLM apps.

What to track

1. Source and ingestion health

Start with the documents that form the application's knowledge base. Track the source identifier, version or modification time, ingestion status, parser errors, document count, and the number of chunks produced. These records help distinguish a retrieval problem from an ingestion problem. If a source stops syncing, the application may appear to have poor reasoning when it is actually searching an incomplete index.

Record whether each document is active, removed, superseded, or awaiting processing. Preserve enough metadata to support access control, filtering, and citations. Useful fields can include title, URL, section heading, owner, publication date, source system, and permission scope. Avoid storing only the text and embedding; without provenance, it becomes difficult to explain or correct an answer.

2. Chunking and indexing quality

Chunking determines what the retriever can find. Track average and maximum chunk size, overlap, empty or nearly empty chunks, duplicate content, and failed embedding jobs. Review representative chunks manually after changing parsing or chunking rules. A chunk that is too short may lack the definition needed to answer a question, while a chunk that is too long may contain several unrelated topics and dilute similarity.

Keep the chunking configuration versioned. When the configuration changes, record which index version was created with it. This makes comparisons possible and allows a rollback if retrieval quality declines.

3. Retrieval quality

For each query, log the retrieved document identifiers, ranks, similarity or relevance scores when available, filters applied, and retrieval latency. Do not treat a high similarity score as proof that the answer is correct. Scores are useful for comparing behavior within the same system, but their meaning can vary across embedding models, indexes, and query types.

Create a small evaluation set containing realistic questions and expected supporting documents. Measure whether the relevant source appears in the top results, whether the result set contains redundant passages, and whether access filters exclude or expose the wrong content. A separate production RAG evaluation checklist can help organize tests for retrieval quality, groundedness, and final answers.

4. Answer quality and citations

Track whether the answer cites retrieved sources, whether the citations actually support the claims, and how often the system responds that the available context is insufficient. Review unsupported claims, partial answers, incorrect source attribution, and answers that combine facts from unrelated documents. If the application serves high-consequence workflows, add a review path rather than relying only on automated scores.

Useful answer-level fields include request identifier, prompt version, model identifier, retrieved context identifiers, output, refusal or abstention status, latency, token usage where available, and user feedback. Protect sensitive content in logs and define retention rules appropriate to the application.

5. Operational and cost signals

Monitor request volume, error rates, time spent in ingestion, embedding, retrieval, and generation, as well as queue depth and timeout frequency. Track usage by route or feature so a change in one workflow does not get hidden inside an application-wide average. Review token consumption and repeated retrieval patterns as part of LLM cost optimization, but do not reduce context blindly: removing relevant evidence can lower cost while making answers less reliable.

Cadence and checkpoints

A maintenance cadence should match how quickly the underlying knowledge changes. A useful starting point is to divide checks into daily, weekly, monthly, and event-driven work.

• Daily: inspect failed ingestion jobs, elevated error rates, latency changes, empty retrieval results, and unusual request volume. Confirm that scheduled source updates completed.
• Weekly: sample real queries, review low-confidence or negatively rated answers, inspect citation support, and check for duplicate or stale content. Add representative failures to the evaluation set.
• Monthly: run the full regression set, compare retrieval and answer metrics with the previous period, review index growth, inspect access-control behavior, and examine token usage. Revisit prompts, chunking, filters, and reranking only when the evidence points to a problem.
• Quarterly: reassess the embedding model, vector database configuration, model selection, retention rules, observability coverage, and disaster-recovery process. Test whether the architecture still fits current traffic and data volume.
• Event-driven: repeat evaluation after changing the model, prompt, parser, chunking strategy, embedding model, vector index, source systems, permissions, or citation format. A change in any one layer can affect the rest of the pipeline.

Use a fixed evaluation set for trend comparisons and a rotating set of recent production examples for freshness. The fixed set shows whether a release regressed known behavior; the rotating set catches new terminology, document structures, and user intents.

Version the important inputs to every evaluation: source snapshot, index version, chunking configuration, embedding model, retrieval settings, prompt, generation model, and application code. Without this information, a metric change is difficult to interpret.

How to interpret changes

Look for patterns across several signals rather than reacting to a single score. If retrieval recall falls after a parser update, inspect extracted text and chunk boundaries before changing the language model. If retrieval is strong but answers are unsupported, review context formatting, system instructions, citation requirements, and the model's tendency to answer beyond the evidence. If answers are accurate but slow, profile each pipeline stage before reducing retrieved context.

Several common patterns are especially useful:

• More empty results: check query normalization, filters, permissions, index availability, and vocabulary changes in the source material.
• Relevant passages rank too low: test chunk boundaries, query rewriting, hybrid retrieval, metadata filters, or reranking.
• Many similar passages are returned: remove duplicates, diversify retrieval, or revise document segmentation.
• Correct context but incorrect answer: inspect prompt instructions, context ordering, answer length, and model behavior on conflicting sources.
• Good evaluation results but poor user feedback: expand the evaluation set. The current tests may not represent the questions users actually ask.
• Stable quality with rising cost: examine repeated queries, unnecessary context, routing opportunities, caching, and model selection without weakening required evidence.

Do not automatically optimize for a single aggregate metric. A system with strong retrieval but weak citations may be unsuitable for a source-grounded workflow. Segment results by query type, tenant, language, source, and access scope when those dimensions matter. Aggregate averages can hide a serious failure affecting only one group.

Prompt changes should follow the same controlled process as code changes. Store prompt versions, run regression tests, compare outputs, and keep a rollback path. The prompt testing framework guide provides a practical workflow for versioning and evaluating prompts.

When to revisit

Revisit this RAG system on a monthly cadence for routine health checks and at least quarterly for architectural review. Increase the frequency when documents change rapidly, users depend on answers for operational decisions, or the application is undergoing active development.

Schedule an immediate review after any of these events:

• a new document source or parser is introduced;
• the embedding or generation model changes;
• chunk size, overlap, metadata, filters, or reranking settings change;
• permissions or tenant boundaries are modified;
• the application receives a new class of user questions;
• latency, error rates, cost, or negative feedback changes materially;
• users report missing, stale, contradictory, or poorly cited answers.

At each review, select a small set of failed and successful examples, trace each one through ingestion, retrieval, prompting, and generation, then record the suspected cause and next test. Update the evaluation set with confirmed failure cases. Keep a short change log covering index versions, prompt revisions, model changes, and source updates.

The goal is not to eliminate every imperfect answer. It is to build a system whose behavior can be inspected, evaluated, corrected, and maintained. Start with one reliable source, a versioned ingestion pipeline, a modest evaluation set, and clear citations. Then expand coverage only as your monitoring shows that each additional source and workflow can be supported safely.