The AI Engineer's Mental Model

The “AI Software Engineer” role is not about training foundation models from scratch. It’s about building reliable software around models that already exist — calling them, grounding them, evaluating them, and shipping features that happen to be powered by them. That reframing changes everything about what you need to know.

So before the fancy techniques, get the core object right: a call to a large language model (LLM) is, at heart, an HTTP request that takes text in and streams text out. Everything else is detail — but the details are where the engineering lives.

Tokens: the unit of everything

Models don’t see characters or words; they see tokens — subword chunks. The word “unbelievable” might be three tokens (un, believ, able). Roughly, one token ≈ 4 characters ≈ ¾ of a word in English.

Tokens matter because they are the unit of three things you constantly manage:

• Cost — you pay per input token and (usually more) per output token.
• Latency — output is generated one token at a time, so longer answers take longer, largely independent of how hard the question is.
• Limits — every model has a maximum context window (input + output tokens it can consider at once). Exceed it and the request fails.

The context window is your working memory

The context window is the single most important constraint in applied AI. The model has no memory between calls — it knows only what you put in the prompt this time. A “conversation” is an illusion you maintain by resending the relevant history on every request.

This is why so much AI engineering is really context engineering: deciding what to put in the window, in what order, within a fixed token budget. Retrieval (next lessons), summarization of old turns, and prompt templates are all tactics for spending that budget well.

Anatomy of a request

A typical chat request has three ingredients:

1. System prompt — durable instructions and persona (“You are a support agent. Answer only from the provided policy.”).
2. Messages — the conversation so far, as alternating user/assistant turns.
3. Parameters — knobs like temperature (higher = more random/creative, lower = more deterministic) and max_tokens (a cap on the output length).

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Answer only from the context. If unsure, say so."},
        {"role": "user", "content": question},
    ],
    temperature=0.2,
)

Determinism is not guaranteed

The same prompt can produce different answers. That non-determinism is a feature for creative work and a hazard for anything you need to test. Lowering temperature reduces variation but never fully eliminates it, which is exactly why evaluation (a later lesson) becomes a first-class engineering concern rather than an afterthought.

The mental model to carry forward

An LLM is a stateless, probabilistic, token-limited function from text to text. Your job as an AI engineer is to build deterministic, reliable systems around that unreliable core — supplying it the right context, constraining its outputs, and measuring its quality.

Hold that sentence in mind. Embeddings, retrieval, serving, and evaluation are all just ways of managing one of those four properties.

Pattern recognition

Variations

Worked problems

def invoice_context(user_id, invoice_id, db):
    invoice = db.get_invoice_for_user(user_id=user_id, invoice_id=invoice_id)
    if invoice is None:
        raise PermissionError("invoice not found for user")
    return {
        "invoice_id": invoice.id,
        "line_items": invoice.line_items,
        "currency": invoice.currency,
    }

def build_messages(profile, conversation_summary, docs, question):
    context = "\n\n".join(
        f"<doc id='{doc['id']}'>\n{doc['text']}\n</doc>" for doc in docs
    )
    return [
        {
            "role": "system",
            "content": (
                "You are a product assistant. Answer only from supplied profile, "
                "conversation summary, and docs. Cite doc IDs when using docs."
            ),
        },
        {
            "role": "user",
            "content": (
                f"<profile>\n{profile}\n</profile>\n\n"
                f"<summary>\n{conversation_summary}\n</summary>\n\n"
                f"<docs>\n{context}\n</docs>\n\n"
                f"Question: {question}"
            ),
        },
    ]

def next_chat_messages(store, conversation_id, new_user_message):
    history = store.recent_turns(conversation_id, limit=8)
    summary = store.summary(conversation_id)
    messages = [{"role": "system", "content": "Be concise and follow user constraints."}]
    if summary:
        messages.append({"role": "user", "content": f"Conversation summary:\n{summary}"})
    messages.extend(history)
    messages.append({"role": "user", "content": new_user_message})
    return messages

The pitfall to avoid