APIs, Load Balancing & Rate Limiting

The API is the front door of the system. It defines what clients can ask for, how failures are retried, how traffic spreads across instances, and how abuse is contained. A clean backend with a sloppy API still becomes an unreliable product.

Good API design is mostly about contracts: stable resource names, predictable pagination, safe retries, explicit limits, and load balancers that only send work to healthy capacity.

The mental model

Think of the request path as three contracts in a row:

client contract  ->  traffic contract  ->  fairness contract
API shape        ->  load balancer      ->  rate limiter
resources/errors ->  health/routing     ->  quotas/backpressure

The API tells clients how to behave. The load balancer decides where each request runs. The rate limiter decides whether the request is allowed to run at all.

REST vs. gRPC vs. GraphQL

A common architecture uses REST at the public edge, gRPC internally, and GraphQL only where client flexibility is worth the operational cost.

Idempotency and pagination

Retries are inevitable. Networks fail after the server succeeds but before the client receives the response. Without idempotency, a retry can create two orders, two payments, or two jobs.

Make unsafe operations retry-safe with an idempotency key:

POST /orders
Idempotency-Key: user-123:create-cart-456

If the same key arrives again, return the original result instead of creating a
second order.

Pagination has the same contract problem. Offset pagination is easy but unstable under inserts: page 2 can skip or duplicate items when new rows arrive. Cursor pagination uses an opaque token tied to the last item seen.

GET /events?limit=50&cursor=eyJsYXN0X2lkIjoi...

The cursor should encode a stable sort key such as (created_at, id), not a page number.

Load balancing

A load balancer spreads traffic across healthy instances.

Common algorithms:

• Round robin — simple rotation; good when requests cost about the same.
• Least connections / least requests — better when request durations vary.
• Weighted routing — send more traffic to larger instances or canaries.
• Consistent hashing — keep the same key on the same backend, useful for sticky caches or long-lived sessions.

Health checks matter as much as the algorithm. A load balancer that routes to a stuck process is an outage amplifier.

Rate limiting

Rate limiting protects shared capacity. The main algorithms:

Token bucket is the interview default because it allows small bursts without letting a client exceed its long-term rate.

Pattern recognition

Variations

Worked problems

In API scenarios, define the client-visible behavior before jumping into servers.

POST /v1/events
Idempotency-Key: tenant-42:device-9:event-abc
Body: { "occurred_at": "2026-07-10T12:00:00Z", "type": "click", ... }

201 Created on first insert
200 OK with the same event id on retry with the same key
409 Conflict if the same key is reused with a different body

GET /v1/events?tenant_id=42&limit=100&cursor=<opaque>
Response: { "items": [...], "next_cursor": "..." }

import time

class TokenBucket:
    def __init__(self, rate_per_sec: float, burst: int):
        self.rate = rate_per_sec
        self.capacity = burst
        self.tokens = burst
        self.updated_at = time.monotonic()

    def allow(self) -> tuple[bool, float]:
        now = time.monotonic()
        elapsed = now - self.updated_at
        self.updated_at = now
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)

        if self.tokens >= 1:
            self.tokens -= 1
            return True, 0.0

        wait_seconds = (1 - self.tokens) / self.rate
        return False, wait_seconds

bucket = TokenBucket(rate_per_sec=100 / 60, burst=20)

The classic pitfall