Designing AI-Powered Systems
Put it together: a production RAG/inference architecture with cost, latency, and failure modes made explicit.
This is where the whole path converges. Designing a system around an LLM reuses every fundamental — load balancing, caching, queues, replication — but adds constraints that traditional services never had: calls are slow (seconds, not milliseconds), expensive (per token), non-deterministic, and dependent on a third party you don’t control. A senior AI engineer designs for those constraints from the start.
A worked example: production question-answering
Imagine the interview prompt: “Design a system that answers employee questions over the company’s internal documents.” That’s RAG — but productionizing it is the real exercise. Here’s the shape of the flow you’d design.
User question
“What is our refund window for damaged items?”
Embed the question
text → [0.11, -0.83, 0.02, … ] (a vector)
Retrieve top-k chunks
vector search → 3 most similar policy snippets
Assemble the prompt
system + retrieved context + question
Generate
LLM reads context, writes an answer
Grounded answer + citations
“Damaged items can be returned within 30 days…” [policy §4]
The two pipelines
Split the system cleanly, exactly as in the RAG lesson:
Ingestion pipeline (offline, event-driven). When a document is added or changed, a queue kicks off a worker that chunks it, calls the embedding API, and upserts vectors into the vector database. Using a queue here means a bulk import of 100,000 documents smooths into a steady stream instead of melting your embedding quota.
Query pipeline (online, latency-sensitive). A stateless API service embeds the question, queries the vector DB for top-k chunks, assembles the prompt, calls the LLM, and streams the answer back with citations.
Designing for the AI-specific constraints
- Latency — model calls dominate the budget. Stream tokens so the user sees words immediately; the p99 that matters is time-to-first-token, not total time. Retrieve and prompt-build in parallel where you can.
- Cost — cache aggressively. An exact-match cache catches repeated questions; a semantic cache (embed the question, look for a near-duplicate) catches paraphrases. Route easy queries to a small model and reserve the expensive model for hard ones.
- Reliability — the LLM provider will have latency spikes and outages. Build in timeouts, retries with backoff, and a fallback (a smaller/secondary model, or a graceful “I can’t answer right now”). Never let a hung upstream call hang your service — use circuit breakers.
- Correctness — apply the guardrails from the AI module: validate structured output, check that answers are grounded in retrieved context, and log every request for offline evaluation.
Capacity and cost estimation
Interviewers love a back-of-the-envelope. Suppose 10,000 questions/day, ~2,000 prompt tokens and ~300 output tokens each. That’s ~23M tokens/day. At a small model’s pricing, that’s dollars, not thousands of dollars — but at 10M questions/day with a frontier model, it’s a serious line item. Doing this arithmetic out loud is what justifies the caching and model-routing you just proposed, and it’s a strong seniority signal.
Observability is non-negotiable
Because the model is a black box, you instrument everything around it: log prompts, retrieved chunks, model, token counts, latency, and cost per request; track hit rates, grounding-failure rates, and user feedback. When quality regresses — and it will, silently, the day a prompt or model version changes — this telemetry is the only way you’ll know and the only way you’ll fix it.
The through-line
Traditional system design keeps data consistent, fast, and available. AI system design adds a slow, costly, unpredictable dependency and asks you to keep the experience fast, cheap, and trustworthy anyway. The primitives are the same; the discipline is applying them with the model’s quirks firmly in mind.
Pattern recognition
Variations
Worked problems
AI design scenarios are system-design scenarios with a slow, costly, probabilistic component. Budget the model like any other dependency.
Budget a RAG answer path
A company assistant serves 100,000 questions/day. Each query embeds the question, retrieves chunks, builds a 2,000-token prompt, and generates a 400-token answer. The product wants first token under 2 seconds at p95. Estimate cost pressure and lay out a latency budget.
Approach. Separate daily token cost from per-request latency. The generation call dominates both; retrieval optimizations help, but model routing and caching usually move the business metrics more.
Show solution
Daily tokens:
input: 100k × 2,000 = 200M input tokens/day
output: 100k × 400 = 40M output tokens/dayAt illustrative prices of $0.50/M input tokens and $2.00/M output tokens:
input cost = 200 × $0.50 = $100/day
output cost = 40 × $2.00 = $80/day
total model tokens ~= $180/day before embeddings/rerankingLatency budget for first token:
| Step | Target |
|---|---|
| Auth and request parsing | 50 ms |
| Question embedding | 100 ms |
| Vector search | 150 ms |
| Rerank/top-k filtering | 200 ms |
| Prompt assembly | 50 ms |
| LLM time-to-first-token | 1,200 ms |
| Buffer/network | 250 ms |
That totals about 2 seconds. The model dominates, so the design should stream immediately after first token, cache repeated questions, route easy queries to a smaller model, and keep retrieved chunks tight to avoid paying for irrelevant context.
If 20% of questions hit an exact or semantic cache, token cost drops by roughly 20% and latency for those requests becomes a cache lookup plus streaming the saved answer.
Design model-provider failover
Your app depends on one LLM provider. Twice this month, provider latency spiked to 10 seconds and your API threads piled up. Design provider failover without making answers unsafe or wildly inconsistent.
Approach. Treat the model provider like any unreliable dependency, but account for model-specific differences: prompts, output schemas, safety behavior, and quality.
Show solution
A robust path:
request
-> model router
-> primary provider with strict timeout
-> circuit breaker opens on high timeout/error rate
-> secondary provider or smaller fallback model
-> output validator
-> response with provider/model/version loggedKey design points:
| Concern | Design response |
|---|---|
| Latency spike | Timeout below user deadline; fail fast when breaker is open |
| Provider outage | Route to secondary provider for eligible requests |
| Output shape | Validate JSON/schema before returning |
| Quality mismatch | Maintain eval sets per provider/model and limit fallback scope |
| Cost jump | Track fallback rate and per-provider token spend |
Not every request should fail over blindly. A regulated answer may require the primary model and current guardrails; in that case graceful degradation is better: “I cannot answer that right now.” For low-risk summarization, a secondary model is reasonable.
The breaker metric is product-facing: if fallback rate rises, you may be serving lower-quality answers even while uptime looks fine.
Cache LLM responses safely
A support chatbot receives many repeated questions such as password reset steps, but some answers depend on the user’s plan and private account state. Design an LLM response cache that saves cost without leaking data or serving stale policy.
Approach. Cache only when the answer is safe to reuse. The cache key must include prompt version, model version, retrieved context, and permission scope — not just the user’s text.
Show solution
Use two layers:
| Layer | Key | Fit |
|---|---|---|
| Exact cache | normalized question + prompt version + model + retrieved doc versions + permission scope | Safe repeated FAQ-style questions |
| Semantic cache | embedding similarity plus same safety/version filters | Paraphrases of public or tenant-scoped questions |
Never cache across permission boundaries. If two users have different document access, their cache scope differs even if the question text is identical.
A safe key includes:
tenant_id or public_scope
user_permission_hash
normalized_question or semantic_cluster_id
retrieved_document_ids_and_versions
prompt_template_version
model_name_and_version
safety_policy_versionInvalidation follows the knowledge base: when a source document changes, bump its version so old answers no longer match. For private account-specific answers, prefer caching intermediate retrieval or tool results with short TTLs over caching the final natural-language answer.
If an average cached answer is 4 KB, one million exact cached answers is only about 4 GB raw before overhead. The hard part is not memory; it is correctness and privacy.