Observability & Reliability
Metrics, logs, and traces; SLOs and error budgets; and the failure patterns (timeouts, retries, circuit breakers) that keep systems up.
A system is reliable only if you can see what it is doing and shape how it fails. Observability gives you the evidence: metrics, logs, and traces. Reliability patterns turn that evidence into behavior: timeouts, retries, circuit breakers, bulkheads, and graceful degradation.
The senior move is to define reliability before the outage. What matters to users? How much failure is acceptable? What does the system do when a dependency is slow, wrong, or unavailable?
The mental model
Observability connects a user promise to machine signals.
user promise -> SLI -> SLO -> alert -> mitigation
"checkout works" -> successful checkouts / attempts -> 99.9% monthlyReliability engineering starts with the promise, not the dashboard. If the chart cannot explain user pain or protect an error budget, it is decoration.
The three pillars
| Pillar | What it answers | Example |
|---|---|---|
| Metrics | What is happening over time? | request rate, p95 latency, error rate, queue depth |
| Logs | What happened in this specific event? | structured error with user id, request id, dependency status |
| Traces | Where did this request spend time? | API -> auth -> database -> payment provider spans |
Use all three together. Metrics tell you something is wrong. Traces tell you where. Logs tell you why this particular request failed.
SLI, SLO, and error budget
| Metrics | Logs | Traces | |
|---|---|---|---|
| Shape | numeric time series | discrete events | causal spans |
| Answers | is it broken? | what happened? | where / why slow? |
| Cardinality | low | high | high |
| Cost | cheap | can explode | sampled |
An SLI is the measurement. An SLO is the target. The error budget is how much unreliability you are allowed before you slow down risky changes.
| Term | Example |
|---|---|
| SLI | Fraction of checkout requests that complete successfully under 800 ms |
| SLO | 99.9% of checkout requests meet that SLI over 30 days |
| Error budget | 0.1% of requests may fail or be too slow |
Time-based intuition helps: 99.9% monthly availability allows about 43 minutes of badness per 30 days. 99.99% allows about 4.3 minutes. Each extra nine is an engineering and product commitment, not a slogan.
Reliability patterns
| Pattern | What it prevents | Key detail |
|---|---|---|
| Timeout | Hanging forever on a dependency | Set shorter than the user-visible deadline |
| Retry with backoff + jitter | Transient failures | Retry only safe/idempotent operations |
| Circuit breaker | Repeated calls to a failing dependency | Fail fast while probing for recovery |
| Bulkhead | One dependency consuming all resources | Separate pools/queues per dependency or tenant |
| Graceful degradation | Total failure when optional features fail | Return core experience without recommendations, avatars, etc. |
Retries deserve suspicion. Retrying a slow dependency can multiply traffic into an outage. Use budgets: if the request deadline is 500 ms, three 500 ms retries are not a reliability strategy.
Pattern recognition
Variations
Worked problems
Reliability scenarios are about evidence and blast radius. Avoid vague answers like “add monitoring”; say exactly what you measure and what the system does.
Define SLOs for checkout
A checkout service takes payment and creates an order. Product asks for “five nines” because checkout is important. Define practical SLIs/SLOs and explain how the error budget guides engineering work.
Approach. Tie the SLO to user-visible success, not internal uptime. Use a latency threshold and a correctness threshold, then calculate the budget.
Show solution
Good SLIs:
| SLI | Why it matters |
|---|---|
| Successful checkout attempts / valid checkout attempts | Captures user-visible failure |
| p95 or p99 checkout latency under 1.5 sec | Captures slow-but-not-failed pain |
| Duplicate charge rate | Captures correctness, not just availability |
A practical initial SLO might be:
99.9% of valid checkout attempts complete successfully within 1.5 seconds over 30 days
0 duplicate charges caused by our systemFor 10 million monthly checkouts, a 99.9% success SLO allows 10,000 bad attempts before the budget is exhausted. That sounds large, so you may choose 99.95% or 99.99% after looking at user harm and engineering cost.
If the service burns half its monthly error budget in one day, freeze risky launches and prioritize reliability work. If it has months of unused budget, the team may be over-investing in reliability relative to product speed.
Add a circuit breaker to a flaky dependency
Your recommendation service calls a third-party ranking API. When that API slows or times out, your own p99 latency spikes and worker threads pile up. Design the failure behavior and sketch the circuit breaker logic.
Approach. Put a strict timeout around the call, track recent failures, open the circuit after a threshold, and return a fallback while probing recovery.
Show solution
import time
class CircuitBreaker:
def __init__(self, max_failures=5, reset_after=30):
self.max_failures = max_failures
self.reset_after = reset_after
self.failures = 0
self.opened_at = None
def allow(self):
if self.opened_at is None:
return True
return time.monotonic() - self.opened_at >= self.reset_after
def record_success(self):
self.failures = 0
self.opened_at = None
def record_failure(self):
self.failures += 1
if self.failures >= self.max_failures:
self.opened_at = time.monotonic()
def get_recommendations(user_id, breaker):
if not breaker.allow():
return cached_or_popular_recommendations(user_id)
try:
result = call_ranker(user_id, timeout_ms=150)
except TimeoutError:
breaker.record_failure()
return cached_or_popular_recommendations(user_id)
breaker.record_success()
return resultThe real design details:
- The timeout must fit inside the page’s total latency budget.
- The fallback should be acceptable product behavior, such as cached or popular items.
- The breaker should expose metrics: open/closed state, failures, fallback rate.
- Use jittered retries only if the operation is safe and the deadline allows it.
This prevents a slow optional dependency from consuming all worker capacity.
Debug a p99 latency spike
At 10:05, p99 latency for POST /orders jumps from 700 ms to 4 sec. Error rate rises slightly. CPU is normal, database latency is normal, and request volume is unchanged. How do you investigate, and what fix do you try first?
Approach. Start from service-level metrics, then use traces to find the slow span. Logs confirm whether the slow path correlates with errors, tenants, or a recent deploy.
Show solution
Investigation path:
1. Metrics: confirm latency spike, error rate, traffic, saturation.
2. Break down by endpoint, region, tenant, and status code.
3. Traces: compare a normal order trace with a slow p99 trace.
4. Logs: inspect request ids from slow traces for dependency errors/timeouts.
5. Change history: check deploys, config changes, dependency incidents.Given CPU and database are normal, traces likely point to an external span: payment provider, tax calculation, fraud check, or shipping quote. If slow traces show payment.authorize taking 3.5 sec, the first mitigation is a tighter timeout plus controlled retry/fallback behavior that fits the checkout contract.
Do not blindly add instances. The app tier is not saturated. More instances may increase concurrent calls to the slow provider and make the incident worse.