Scaling Fundamentals
Latency vs. throughput, vertical vs. horizontal scaling, and the numbers every engineer should know by heart.
System design interviews and real architecture both start from the same place: a shared vocabulary and a feel for the numbers. Before any fancy diagram, you need to reason clearly about what “fast” and “big” actually mean.
Latency vs. throughput
These two get conflated constantly, so nail the distinction:
- Latency is how long one request takes (e.g. 200 ms).
- Throughput is how many requests you handle per second (e.g. 5,000 rps).
They’re independent. A system can be low-latency but low-throughput (a fast single-lane road) or high-latency but high-throughput (a slow but very wide highway). You optimize them with different techniques, and a good design states targets for both.
And don’t reason about latency with averages — reason with percentiles. “p99 latency is 800 ms” means 1% of requests are that slow. At scale, that 1% is millions of users, so p99 and p999 are what teams actually get paged about.
Vertical vs. horizontal scaling
- Vertical (“scale up”) — buy a bigger machine. Simple, but there’s a ceiling and a single point of failure.
- Horizontal (“scale out”) — add more machines behind a load balancer. Effectively unlimited, and it gives you redundancy, but it forces you to confront distributed-systems problems: shared state, consistency, and coordination.
Serious systems scale horizontally, which is why the rest of this module is really about coordinating many machines.
Stateless services scale; state is the hard part
| Vertical (bigger box) | Horizontal (more boxes) | |
|---|---|---|
| Ceiling | hardware limit | near-unbounded |
| Complexity | trivial | LB, coordination |
| Fault tolerance | single point | redundant |
| Cost curve | super-linear | roughly linear |
A stateless service keeps no per-user data in its own memory between requests, so any instance can handle any request and you can add or remove instances freely. The moment a service holds state, horizontal scaling gets hard.
The standard move is to push state down into systems built to be shared: databases, caches, and object stores. Your application tier stays stateless and trivially scalable; the stateful tier is handled by specialized infrastructure.
Databases under load
Two levers dominate database scaling:
- Replication — copy data to multiple nodes. A common setup is one primary for writes and several read replicas, which scales reads and adds failover. The catch is replication lag: a replica may briefly serve slightly stale data (eventual consistency).
- Sharding (partitioning) — split data across nodes by a key (e.g. user ID), so each node holds a slice. This scales writes and storage, but cross-shard queries and choosing a good shard key are genuinely hard.
The CAP trade-off, in one breath
When the network partitions (and at scale, it will), a distributed store must choose between consistency (every read sees the latest write) and availability (every request gets an answer). You can’t have both during a partition. Most user-facing systems lean toward availability plus eventual consistency; systems handling money lean toward strong consistency.
Numbers worth memorizing
Rough orders of magnitude that make estimation possible:
| Operation | Time |
|---|---|
| L1/L2 cache reference | ~1 ns |
| Main memory reference | ~100 ns |
| SSD random read | ~100 µs |
| Round trip within a datacenter | ~500 µs |
| Disk seek (spinning) | ~10 ms |
| Round trip across continents | ~100–150 ms |
The headline: memory is ~1000× faster than disk, and a cross-continent round trip dwarfs everything. That single fact motivates caches, CDNs, and keeping data close to users — the subjects of the lessons that follow.
Pattern recognition
Variations
Worked problems
The first pass in system design is usually not a database diagram. It is naming what kind of scaling problem you are actually solving.
Move from one box to a fleet
A web app runs on one large VM. At 1,000 requests/sec, CPU sits around 85% and p95 latency is 900 ms. The next launch is expected to peak at 4,000 requests/sec. Sessions are stored in process memory. What do you change first?
Approach. The app tier is the bottleneck, but horizontal scaling only works after request state moves out of the instance. Estimate instance count with headroom, then add a load balancer and shared session strategy.
Show solution
Treat the current VM as unsafe above about 800 requests/sec; 85% CPU leaves little room for spikes or garbage collection pauses.
4,000 peak rps / 800 safe rps per instance = 5 instances
Add N+2 headroom for deploys/failures: run 7 instancesThe design change:
client -> load balancer -> stateless app instances
-> shared session store or signed cookies
-> database/cache/object storeMove sessions to Redis, a database-backed session store, or signed encrypted cookies. Add health checks so unhealthy instances leave rotation. Once the app tier is stateless, autoscaling and rolling deploys become straightforward.
Do not start with sharding the database. The stated bottleneck is app CPU and in-process state; solve that before adding distributed data complexity.
Find the real bottleneck
An API has average latency of 120 ms but p99 latency of 2.5 sec during peak. App CPU is 35%, database CPU is 92%, and the database connection pool is often full. A teammate suggests doubling the number of app servers. Do you agree?
Approach. Low app CPU plus saturated database means the app tier is waiting, not working. More app servers may increase pressure on the constrained database.
Show solution
Do not double app servers as the primary fix. The evidence points to the database:
| Signal | Interpretation |
|---|---|
| App CPU 35% | App instances are not compute-bound |
| DB CPU 92% | Database is close to saturation |
| Full connection pool | Requests are queueing for database access |
| High p99, normal average | Tail latency from contention, not every request slow |
Better first moves:
- Identify the slow queries and add targeted indexes.
- Cache hot read-heavy endpoints if staleness is acceptable.
- Split read traffic to replicas if reads dominate.
- Put slow noncritical work behind a queue.
- Cap app concurrency so the database is not flooded during spikes.
Adding app servers can help only after the database bottleneck is relieved. Before that, it risks turning a slow tail into a full outage.
Externalize state before scaling uploads
An image upload service stores the uploaded file on local disk, writes metadata to the database, then a background thread resizes the local file. You add more app instances behind a load balancer and suddenly resizes fail intermittently. Why, and what architecture fixes it?
Approach. Local disk made the service stateful. A later request or worker may land on a different instance that cannot see the file. Push the blob to shared object storage and move slow resizing to a queue-backed worker.
Show solution
The failure is instance-local state:
upload handled by instance A -> file saved on A's disk
resize job handled by instance B -> file not foundA safer architecture:
client -> upload API -> object storage (original)
-> metadata DB row
-> queue message: resize image_id
worker -> object storage original -> thumbnails -> object storage
-> update metadata statusNow any app instance can accept uploads, and any worker can process resizes. The request path returns after durable upload and enqueue, not after CPU-heavy image processing. If workers fall behind, the queue depth grows instead of user requests timing out.
This is the general rule from the lesson: stateless services scale horizontally; state belongs in systems designed to be shared.