Retrieval-Augmented Generation

Language models have two well-known failure modes: they don’t know about your private or recent data, and they will confidently hallucinate an answer when they don’t know. Retrieval-Augmented Generation (RAG) addresses both by fetching relevant facts at query time and handing them to the model as context — so it can read the answer instead of recalling it.

RAG is the workhorse pattern of applied AI. Follow a single question through the pipeline, stage by stage.

The two phases

RAG splits cleanly into an offline phase and an online phase.

Indexing (offline, done once):

1. Load your documents.
2. Chunk them into passages.
3. Embed each chunk and store the vectors in a vector database.

Retrieval + generation (online, per query):

1. Embed the user’s question.
2. Retrieve the top-k most similar chunks.
3. Assemble a prompt: instructions + retrieved context + the question.
4. Generate an answer grounded in that context.
5. Return the answer with citations back to the source chunks.

def answer(question, k=4):
    q_vec = embed(question)
    chunks = vector_db.search(q_vec, k=k)          # retrieve
    context = "\n\n".join(c.text for c in chunks)  # assemble
    prompt = (
        "Answer the question using ONLY the context below. "
        "If the context is insufficient, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return llm(prompt), [c.source for c in chunks]  # generate + cite

Why grounding beats memorizing

Because the facts arrive at query time, RAG gives you three things fine-tuning can’t easily match:

• Freshness — update a document and the next answer reflects it, no retraining.
• Attribution — you can show where each claim came from, which is often a hard product requirement.
• Access control — retrieve only chunks the current user is allowed to see, and the model can only speak to those.

Where RAG quietly breaks

Most “the AI gave a bad answer” bugs are really retrieval bugs, not model bugs. If the right chunk never makes it into the context, no model can save you. The usual suspects:

• Bad chunking splits a fact across two chunks so neither is retrievable.
• Retrieval misses because the question and the answer are worded very differently (semantic gap).
• Context stuffing crams in too many low-relevance chunks, burying the good one and inflating cost.

Common remedies include hybrid search (combining keyword and vector search), re-ranking the retrieved set with a stronger model, and query rewriting to bridge the wording gap.

RAG vs. fine-tuning

A frequent interview question. The clean distinction:

RAG changes what the model knows; fine-tuning changes how the model behaves. Use RAG to inject facts (docs, policies, product data). Use fine-tuning to teach a consistent format, tone, or narrow skill. They are complementary, not competing — many production systems use both.

Pattern recognition

Variations

Worked problems

def chunk_paragraphs(paragraphs, max_chars=2_800, overlap_chars=400):
    chunks = []
    current = ""
    for paragraph in paragraphs:
        if len(current) + len(paragraph) > max_chars and current:
            chunks.append(current.strip())
            current = current[-overlap_chars:] + "\n" + paragraph
        else:
            current += "\n" + paragraph
    if current.strip():
        chunks.append(current.strip())
    return chunks

def missing_support(expected_chunk_ids, retrieved_chunk_ids):
    expected = set(expected_chunk_ids)
    retrieved = set(retrieved_chunk_ids)
    return sorted(expected - retrieved)

def build_grounded_prompt(question, chunks):
    context = "\n\n".join(
        f"[chunk:{c['id']}]\n{c['text']}" for c in chunks
    )
    return (
        "Answer using only the chunks below. Every factual claim must cite "
        "one or more chunk IDs like [chunk:abc]. If the chunks do not support "
        "the answer, say: Cannot determine from provided sources.\n\n"
        f"{context}\n\nQuestion: {question}"
    )

def validate_citations(answer, retrieved_ids):
    cited = {part.split("]")[0] for part in answer.split("[chunk:")[1:]}
    cited = {f"chunk:{x}" for x in cited}
    allowed = {f"chunk:{x}" for x in retrieved_ids}
    return cited <= allowed and bool(cited or "Cannot determine" in answer)

The pitfall to avoid