Caching & the CDN

The fastest request is the one you never make. Caching — keeping a copy of expensive-to-produce data somewhere fast — is the highest-leverage performance technique in systems design, which is why it appears in almost every design interview. It’s also, famously, one of the two hard problems in computer science.

Why caching wins

Recall the numbers from the last lesson: memory is roughly a thousand times faster than disk, and a cross-continent round trip is ~100 ms. A cache turns a slow database query or a remote API call into a memory lookup. For read-heavy workloads — which most systems are — a well-placed cache can absorb the large majority of traffic before it ever touches your database.

Caches live at every layer

Caching isn’t one thing; it’s a strategy applied all the way down the stack:

• Browser / client — cache static assets and API responses locally.
• CDN — a Content Delivery Network stores copies at edge locations physically near users, so a request from Tokyo is served from Tokyo, not from a server in Virginia. Ideal for images, video, JS/CSS, and cacheable API responses.
• Application cache — an in-memory store like Redis or Memcached sitting between your app and the database, holding hot query results, sessions, and computed values.
• Database cache — the database’s own buffer pool keeps recent pages in memory.

The cache-aside pattern

The most common application pattern. The application manages the cache explicitly:

def get_user(user_id):
    cached = cache.get(user_id)
    if cached is not None:
        return cached                  # cache hit
    user = db.query(user_id)           # cache miss → hit the source
    cache.set(user_id, user, ttl=300)  # populate for next time
    return user

Two numbers define its health: the hit rate (fraction served from cache) and the TTL (how long an entry lives before it must be refetched).

Invalidation: the genuinely hard part

Reading from a cache is easy. Keeping it correct is not — that’s the “hard problem.” When the underlying data changes, stale cache entries can serve wrong answers. The main strategies:

• TTL expiry — entries simply expire after N seconds. Dead simple; the cost is a window of staleness you must decide is acceptable.
• Write-through — write to the cache and the database together, so the cache is never stale (at the cost of slower writes).
• Explicit invalidation — delete or update the cache entry when the data changes. Precise, but easy to get wrong when many things touch the same data.

The right question is never “how do I cache this?” but “how stale is this data allowed to be?” A stock price and a user’s display name have wildly different tolerances, and that tolerance dictates the strategy.

Failure modes to name in an interview

• Thundering herd / cache stampede — a popular key expires and thousands of requests hit the database simultaneously to refill it. Mitigate with locks so only one request rebuilds, or by staggering expiry.
• Cold cache — after a restart or deploy, the cache is empty and the database briefly takes full load. Warming strategies help.

Mentioning these unprompted signals that you’ve operated caches, not just read about them.

Pattern recognition

Variations

Worked problems

Caching problems are freshness problems disguised as performance problems. Always say what stale answer is acceptable.

read product -> cache-aside with TTL
update price -> transaction commits in DB
             -> publish ProductUpdated(product_id, version)
             -> delete or replace cache key product:{id}

metadata row stores cache_version = 18
cache key = product:{id}:v18
price update increments version to 19
old key naturally expires; readers use v19 immediately

50,000 requests/sec × 0.2 sec query time = ~10,000 concurrent DB queries

cache hit and fresh -> return
cache hit but soft-expired -> trigger one refresh, return stale
cache miss and lock acquired -> rebuild, set cache, return
cache miss and lock held -> wait briefly or return fallback