Prompting vs. RAG vs. Fine-tuning

When an AI feature is wrong, you have three main levers: change the prompt, retrieve better context, or change the model with fine-tuning. Senior AI engineering is knowing which lever moves the failure you actually have.

The common mistake is treating fine-tuning as a memory upgrade. It is usually the wrong tool for fresh facts, private documents, or anything users expect to cite. Fine-tuning changes behavior; RAG supplies knowledge at request time.

The mental model

Think of an LLM product as a function with three inputs you control:

def ai_feature(user_request, *, prompt_template, retrieved_context, model):
    messages = prompt_template.render(
        request=user_request,
        context=retrieved_context,
    )
    return model.generate(messages)

• Prompting changes the instructions and examples sent on this call.
• RAG changes the facts available on this call.
• Fine-tuning changes the model weights used for every call.

Those levers operate at different timescales. A prompt can ship in minutes. A RAG index can update as documents change. A fine-tuned model needs data curation, training, evaluation, deployment, and rollback.

Prompting: the first lever

Prompting is the cheapest and fastest way to change behavior. Use it when the model already has the capability and you need to specify the task boundary: format, tone, refusal policy, examples, or routing criteria.

Good prompting fixes:

• inconsistent output shape,
• missing constraints,
• ambiguous task definitions,
• weak examples for edge cases,
• accidental instruction/data mixing.

Prompting does not reliably teach new private facts, and it gets expensive if you paste a large manual into every request.

RAG: the knowledge lever

RAG retrieves facts at query time and places them in the context window. Use it when the answer depends on knowledge outside the base model: private docs, customer data, recent policies, product catalogs, or source code.

RAG is strongest when you need:

• freshness without retraining,
• citations and auditability,
• access control per user,
• a clear path to update or delete knowledge.

Its maintenance cost moves into chunking, indexing, retrieval evaluation, and prompt assembly. If retrieval misses, generation quality collapses.

Fine-tuning: the behavior lever

Fine-tuning trains the model on examples so the weights shift toward a desired behavior. Use it when prompting examples are too many, too expensive, or too weak to reliably teach the behavior.

Good fine-tuning targets:

• a narrow output style or schema,
• domain-specific classification,
• tool-call argument patterns,
• repeated transformations with many labeled examples,
• reducing prompt length by baking examples into the model.

High-level PEFT methods like LoRA add small trainable adapter matrices instead of updating every model weight. That makes training cheaper and easier to swap, but it does not remove the need for clean data and evals.

Cost, latency, and maintenance

The levers have different operational shapes:

• Prompting is low setup cost but can increase per-request tokens and become fragile as rules accumulate.
• RAG adds retrieval latency and index maintenance, but keeps facts fresh and auditable.
• Fine-tuning adds data, training, deployment, and rollback work, but can reduce prompt size and improve consistency at inference time.

A mature product often uses all three: prompt for the task, RAG for facts, and a fine-tuned or smaller model for a repeated narrow behavior.

Pattern recognition

Variations

Worked problems

def choose_lever(scenario: str) -> str:
    if scenario == "changing_hr_handbook":
        return "RAG + prompt"
    if scenario == "vendor_email_json":
        return "prompt + schema validation; fine-tune if evals stay weak"
    if scenario == "brand_voice":
        return "prompt with examples first; fine-tune if high volume and examples exist"
    raise ValueError(scenario)

def should_finetune_for_knowledge(fresh: bool, needs_citations: bool, acl: bool) -> bool:
    return not (fresh or needs_citations or acl)

def macro_f1_by_label(confusions):
    scores = []
    for label, counts in confusions.items():
        tp = counts["tp"]
        fp = counts["fp"]
        fn = counts["fn"]
        precision = tp / (tp + fp) if tp + fp else 0.0
        recall = tp / (tp + fn) if tp + fn else 0.0
        f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
        scores.append(f1)
    return sum(scores) / len(scores)

The classic pitfall