The ML / AI Design Interview
The round unique to AI roles: frame the problem, pick data and metrics, and design the system around the model.
The ML / AI design round asks a different question from generic system design: can you turn an ambiguous product goal into a measurable, data-backed, model-aware system that keeps working after launch? The architecture still matters, but the model is not a magic box in the middle. Data, labels, metrics, feedback loops, and failure modes are the core of the round.
A strong answer sounds like an engineer who has shipped ML, not a researcher reciting model names. You frame the product outcome, choose the data and metrics, explain how the model will be trained or prompted, and design the serving and monitoring loop around the places it can fail.
The mental model
Run the same framework every time:
| Step | Question to answer | Senior signal |
|---|---|---|
| Clarify problem | What user outcome are we improving? | You avoid optimizing the wrong metric. |
| Success metric | What offline and online numbers define good? | You separate model quality from product impact. |
| Data and labels | What examples train or evaluate the system? | You know labels are often the bottleneck. |
| Representation | What features, embeddings, context, or retrieval inputs matter? | You connect product data to model behavior. |
| Model choice | What baseline, model family, or LLM strategy is appropriate? | You choose the simplest model that can work. |
| Serving | What latency, cost, freshness, and reliability constraints shape inference? | You can ship it, not just train it. |
| Evaluation | How do we test offline and online before trusting it? | You prevent silent regressions. |
| Monitoring | What drift, quality, abuse, and feedback loops do we watch? | You design for the months after launch. |
| Failure modes | How does it fail, and what is the fallback? | You are honest about risk. |
The best answers move through the framework, then go deep where the prompt is most interesting. A recommendation feed might deep dive ranking and exploration. A RAG assistant might deep dive retrieval quality and hallucination. A moderation classifier might deep dive label policy and false-positive cost.
- Clarify problemwhat user outcome are we improving?
- Success metricoffline quality vs. online impact
- Data & labelsoften the real bottleneck
- Representationfeatures, embeddings, retrieval inputs
- Model choicesimplest model that can work
- Servinglatency, cost, freshness, reliability
- Evaluationoffline + online before trusting it
- Monitoringdrift, abuse, feedback loops
- Failure modeshow it fails and the fallback
How this differs from system design
Generic system design starts with scale and components: clients, services, databases, caches, queues, replication. ML design still needs that, but it adds a second axis: how the system learns and how you know it is right.
| Generic system design asks… | ML / AI design also asks… |
|---|---|
| Can it handle traffic? | Is the model making the right decision? |
| Where is data stored? | How is data labeled, refreshed, and de-biased? |
| What is the p95 latency? | What quality do we trade for that latency? |
| How do we recover from failure? | What is the safe fallback when predictions are uncertain? |
| What do we monitor? | What drift, feedback loops, and bad labels corrupt quality? |
Pattern recognition
Variations
Worked problems
Design a recommendation feed
Design the ranking system for a personalized feed in a short-form learning app. Users open the app, see a feed of lessons, videos, and practice prompts, and can save, skip, complete, or dismiss items.
Approach. This is a recommendation system, so start with the product outcome: help users find useful next content, not merely maximize clicks. Then separate candidate generation from ranking, define labels from user behavior, and talk through serving freshness, exploration, and feedback loops.
Show solution
A strong walkthrough:
| Framework step | Worked answer |
|---|---|
| Clarify problem | “Are we optimizing for engagement, learning completion, retention, or interview readiness? I would avoid pure click-through because sensational items can win while learning quality drops.” |
| Success metric | Online: completion rate, saves, next-day return, downstream practice success. Guardrails: hides, reports, time wasted, diversity of topics. Offline: ranking metrics such as NDCG or recall against historical positive actions. |
| Data and labels | Positive labels: completed, saved, high rating, follow-up practice success. Weak positives: clicked or watched briefly. Negatives: skipped, dismissed, abandoned quickly. Watch for position bias because top-ranked items get more interactions. |
| Representation | User embedding from completed topics, skill level, recency, goals, and weak signals. Item embedding from topic, difficulty, format, text, and historical engagement. Add freshness and prerequisite metadata. |
| Model choice | Start with retrieval candidates from embeddings and rules, then rank with a gradient-boosted model or neural ranker. Keep a simple popularity/prerequisite baseline for cold start. |
| Serving | Precompute user and item embeddings, generate candidates offline or nearline, rank online with fresh context. Cache feed pages but leave room for real-time events like “user just completed heaps.” |
| Evaluation | Backtest on held-out interactions, then A/B test against the current feed. Segment by new users, advanced users, and underrepresented topics. |
| Monitoring | Track metric drift, topic collapse, repeated items, cold-start quality, latency, and whether the system over-optimizes easy content. |
| Failure modes | Filter bubbles, clickbait learning paths, stale recommendations, unfair exposure for new content. Add exploration slots and diversity constraints. |
The trade-off to name: a more complex ranker may lift engagement, but it also makes explanations, debugging, and feedback loops harder. For a learning product, I would reserve part of the feed for pedagogically important items even if they are not predicted to maximize clicks.
Design a RAG-based support assistant
Design an AI support assistant for a SaaS product. It should answer customer questions using the company’s docs and support history, cite sources, and hand off to a human when it is uncertain.
Approach. This is not just chat. It is retrieval, generation, evaluation, and risk management. Split the design into offline indexing and online answering, then define what “good” means: grounded, helpful, safe, and fast enough.
Show solution
A strong walkthrough:
| Framework step | Worked answer |
|---|---|
| Clarify problem | “Which channels matter: in-app chat, email, or internal agent assist? Are answers allowed to take actions, or only provide guidance? What sources are authoritative?” |
| Success metric | Online: deflection with no reopen, customer satisfaction, human escalation quality, time to resolution. Guardrails: hallucination rate, unsafe action suggestions, citation coverage. Offline: answer correctness, groundedness, retrieval recall. |
| Data and labels | Docs, release notes, help-center articles, resolved tickets, product metadata. Labels from expert-written eval sets, support-agent judgments, and customer outcome signals. Keep private tickets permissioned and scrub sensitive data. |
| Representation | Chunk docs by semantic section, store embeddings plus metadata: product area, version, customer tier, freshness, permissions. Query representation may include account context if allowed. |
| Model choice | Use retrieval plus a strong hosted LLM first. Fine-tuning is not the first lever; improve retrieval, prompts, and evals before training. Use smaller models for classification tasks like route-to-human. |
| Serving | Online path: classify intent and risk, retrieve top chunks with hybrid search, rerank, build a grounded prompt, stream answer with citations, and enforce a confidence threshold for handoff. Cache common retrievals and answers where safe. |
| Evaluation | Maintain a golden set of support questions with expected sources and answer criteria. Measure retrieval recall separately from generation quality so you know whether failures come from missing context or bad synthesis. Run A/B tests carefully because bad answers can harm customers. |
| Monitoring | Track unanswered questions, escalation rates, hallucination reports, citation clicks, stale-source usage, cost per conversation, latency, and drift after product launches. |
| Failure modes | Hallucinated policies, stale docs, permission leaks, overconfident answers, prompt injection in retrieved content, and low-quality handoffs. Fallback: cite uncertainty and route with a concise summary to a human. |
The senior trade-off is confidence vs. coverage. You can answer more questions by lowering the threshold, but support quality may drop. I would launch with a high handoff rate, learn from escalations, then expand automation where the evals are strong.
Design a content-moderation classifier
Design a system that flags user-generated posts for hate, harassment, sexual content, and self-harm risk. Some cases should be auto-blocked; others should go to human review.
Approach. Moderation is a classification problem where policy and thresholds matter as much as model architecture. Define categories, label policy, severity, false-positive and false-negative costs, and the review loop.
Show solution
A strong walkthrough:
| Framework step | Worked answer |
|---|---|
| Clarify problem | “What content types: text only, images, video, links? What policies define each class? What actions are allowed: warn, downrank, block, escalate, or notify emergency resources?” |
| Success metric | Per-class precision and recall, calibrated risk scores, review SLA, appeal overturn rate, user harm reports. Optimize thresholds differently by class because missing self-harm risk and incorrectly blocking benign speech have different costs. |
| Data and labels | Historical moderated content, appeals, policy-team examples, synthetic edge cases reviewed by humans. Labels need clear guidelines and inter-rater agreement; ambiguous policy creates noisy training data. |
| Representation | Text features or transformer embeddings, conversation context, user/report metadata, language, media embeddings if multimodal. Include policy version so labels remain interpretable after rules change. |
| Model choice | Start with a strong pretrained classifier fine-tuned on policy labels. Use separate heads or thresholds per class. Consider rules for known illegal content and human review for uncertain high-impact cases. |
| Serving | Score posts synchronously if they are public immediately; otherwise queue for async review. Apply thresholds: low risk allow, medium risk review, high confidence severe risk block or escalate. Keep latency budget explicit for posting flow. |
| Evaluation | Use stratified eval sets with rare severe classes, adversarial examples, and multilingual slices. Track calibration so a score of 0.9 actually means high risk. Evaluate policy changes before rollout. |
| Monitoring | Drift by language, slang, coordinated abuse, false positives from appeals, reviewer backlog, and threshold impact on communities. Feed reviewed decisions back into training with delay and quality checks. |
| Failure modes | Bias against dialects, adversarial spelling, policy ambiguity, reviewer overload, and feedback loops where only flagged content gets labeled. Use sampling of allowed content to estimate false negatives. |
The answer should be explicit that this is a socio-technical system. A model can prioritize and automate obvious cases, but policy, human review, appeals, and monitoring are part of the design, not operational afterthoughts.
Closing the loop
The ML design interviewer is listening for disciplined uncertainty. You will not know the perfect model. You can know what metric matters, what data would prove progress, what fallback protects users, and what you would monitor after launch. That is the senior signal.