The design order, estimation numbers, building-block rules and failure-containment patterns on one page.
In the print dialog, choose "Save as PDF" as the destination.
The order
1. Clarify - users, what must never be wrong, SLO
2. Estimate - requests per second, storage, bandwidth, peak
3. Sketch - smallest design that meets the numbers
4. Deep-dive - the risky part: data store, queue, rollout
5. Failures - slow, down, wrong, overloaded, for every box
6. Trade-offs - say what you gave up
Numbers to remember
seconds per day ~ 100,000 (86,400)
1M requests/day ~ 10 per second
100M requests/day ~ 1,000 per second
peak versus average ~ 3x unless known
memory read ~100 ns | SSD read ~100 us | same-region RTT ~1 ms | cross-continent RTT 100-200 ms
99.9% = 43 min/month | 99.99% = 4.3 min/month | 99.999% = 26 s/month
Building blocks in one line each
Cache: jitter TTLs, coalesce refills, serve stale, warm before cutover, size the store for the cold case.
Queue: assume at-least-once, make consumers idempotent, bound the queue, watch age not length.
Retry: timeout shorter than the caller, exponential backoff with jitter, retry at one layer, use a budget.
Idempotency: client key, stored with the result in one transaction.
Consistency: pick per feature; fix stale reads with read-your-writes; alert on replication lag.
Quorum: majority of an odd number; 3 tolerates 1, 5 tolerates 2; spread across zones.
Fencing: the resource rejects a lower token; a lease alone is not enough.
Time: monotonic for durations; never order by wall-clock across machines; handle negative differences.
Containing failure
failure domain what one cause takes down together (process, host, zone, region, change)
cell full independent copy for a slice of users, thin router, hard size limit
bulkhead separate pools per dependency
shuffle shard random worker subset per customer
canary smallest stage with a stop signal and bake time
fail static keep last known good on bad input
fail open/closed decide per check, write it down, test it
Overload
token bucket steady refill, bounded burst
concurrency limit cap in-flight work (protects slow paths)
shed by priority batch first, critical last
signals 429 or 503 with Retry-After; clients back off with jitter
autoscaling slow; limits hold the line meanwhile
Multi-region
RTO = time down RPO = data lost, in time
backup/restore < pilot light < warm standby < active-active
survivors must carry the lost share (N regions need headroom)
two DNS providers; drill the failover; recovery tools off the failing system
Retry arithmetic
layers retrying 3 x 3 x 3 = 27 calls per user request at the bottom
fix: retry at one layer, jittered backoff, budget of ~10% of traffic
Failover checks
# who is writable? exactly one should answer false for recovery
psql -h NODE -Atc "select pg_is_in_recovery()"# replication lag in seconds on a replica
psql -Atc "select extract(epoch from now()-pg_last_xact_replay_timestamp())"
Change safety checklist
Can it be previewed on real data?
Smallest first stage and stop signal?
Previous good state and rollback time?
What does the consumer do with garbage?
Does the fixer depend on the broken thing?
Official references
Read the vendor and project documentation for the full, current detail.
