Building a production RAG pipeline: chunking, embeddings and retrieval evaluation

Retrieval-augmented generation (RAG) looks simple in a demo: put documents in a vector database, retrieve the top five chunks, paste them into a prompt. In production that demo usually breaks within weeks. Answers cite the wrong policy version, tables come back as broken text, and nobody can say whether last week's "improvement" helped or hurt. The model is rarely the problem. Retrieval is. This guide covers how to build a RAG pipeline that holds up in production.

This article walks through the three decisions that matter most (chunking, embeddings and indexing, and evaluation) and the engineering habits that make a RAG system stay good after launch.

Start from the questions, not the documents

Before writing any pipeline code, collect 100–300 real questions from the people who will use the system: support agents, analysts, compliance officers. For each one, write down which document (and ideally which paragraph) contains the answer. This becomes your golden set. It is the single most valuable asset in the project, because it turns every later decision from opinion into measurement.

The golden set also tells you what kind of system you are building. If most questions are lookups ("what is the margin requirement for product X?"), precise retrieval of short passages matters. If they are synthesis ("summarise how our onboarding rules changed this year"), you need broader context, metadata about dates and versions, and probably more than one retrieval step.

RAG chunking strategy: preserve meaning, not character counts

Chunking is where most quality is lost. Fixed-size splitting (say, 1,000 characters with 200 overlap) is a reasonable baseline, but it cuts sentences, separates headings from their content and shreds tables.

Better strategies, roughly in order of effort:

• Structure-aware splitting. Parse the document format first: Markdown or HTML headings, PDF layout blocks, Word styles. Split on sections and paragraphs, then merge small pieces until you reach a target size (300–800 tokens works for most prose).
• Carry the context with the chunk. Prepend the document title and heading path ("Trading Rules › Margin › Intraday") to each chunk's text before embedding. This one change often gives a noticeable recall gain, because many chunks are meaningless on their own.
• Treat tables and lists specially. Convert tables to Markdown or to one row per line with column names repeated. A row like Product: Gold futures | Initial margin: 8% retrieves far better than a flattened grid of numbers.
• Parent–child retrieval. Index small chunks for precise matching, but return the larger parent section to the model. You get sharp retrieval and enough surrounding context for a correct answer.

Keep metadata on every chunk: source, version, effective date, language, access level. You will need it for filtering, for citations and for permissions.

Embeddings for RAG and indexing

Pick an embedding model by testing it on your golden set, not by leaderboard rank. Multilingual content matters here. If your corpus mixes Persian and English, test cross-lingual retrieval explicitly: a Persian question against English documents, and the other way round. Some strong English models degrade badly on Persian script, normalisation and half-spaces.

Practical points:

• Normalise text consistently before embedding and at query time: Unicode normalisation, Arabic and Persian character variants (ي/ی, ك/ک), digits and whitespace. Mismatched normalisation silently destroys recall.
• Use hybrid search. Combine dense vectors with a lexical index such as BM25. Dense retrieval is good at paraphrase; lexical search is good at product codes, names and exact terms. Fuse the two result lists with reciprocal rank fusion, which needs no score calibration.
• Add a reranker. Retrieve 30–50 candidates cheaply, then rerank them with a cross-encoder that reads query and passage together. Rerankers are slower per item but dramatically improve the top three results, which is what the model actually sees.
• Index choice follows scale. Up to a few million chunks, HNSW in a mature vector store or in PostgreSQL with pgvector is usually enough. Tune ef_search and M against recall on your golden set, not against defaults.
• Version your index. Store which embedding model and chunking configuration produced each index. Changing either means a full re-embed. Plan for blue/green indexes so you can switch and roll back.

Retrieval evaluation: measure retrieval separately from generation

Most teams evaluate only the final answer. That hides where problems come from. Measure in two layers.

Retrieval metrics using your golden set:

• Recall@k: does the correct passage appear in the top k results? This is the ceiling for answer quality. If recall@5 is 60%, no prompt engineering will rescue the other 40%.
• MRR (mean reciprocal rank) or nDCG: how high does the correct passage rank?
• Break results down by question type, language and source. Averages hide the failing segments.

Generation metrics, judged by people and by an LLM grader calibrated against people:

• Faithfulness: is every claim in the answer supported by the retrieved context?
• Answer relevance: does it address the question that was asked?
• Citation correctness: do the cited sources actually contain the claim?

Run the full evaluation on every change to chunking, embeddings, prompts or models, in CI, the same way you run unit tests. A 2% drop in recall that nobody noticed is how RAG systems decay.

Production concerns that are not optional

• Permissions at retrieval time. Filter by the user's access rights inside the search query, not after generation. A model cannot "un-see" a confidential paragraph.
• Freshness. Build incremental ingestion: detect changed documents, re-chunk only what changed, and delete stale chunks. Expose the source date in answers.
• Refusal behaviour. If the best retrieval score is below a threshold, say so and offer an escalation path instead of letting the model improvise.
• Observability. Log the query, retrieved chunk IDs, scores, the prompt and the answer. When a user flags a bad answer, you should be able to replay exactly what happened.
• Feedback loop. Every flagged answer is a candidate for the golden set. The set should grow every month.

A sensible first architecture

For most organisations a robust first version is: structure-aware chunking with heading context, a multilingual embedding model, hybrid retrieval with reciprocal rank fusion, a cross-encoder reranker, a prompt that requires citations, and an evaluation suite that runs on every change. It is not exotic, and it does not need agents or graph databases on day one. It does need discipline about data and measurement.

RAG quality is mostly an information-retrieval problem with a language model at the end. Treat it that way, invest early in the golden set, and the system will improve over time instead of slowly drifting.