System design deep dive · HLD
Design a Content Delivery Network (CDN)
A CDN pushes content and compute to hundreds of points of presence so requests never have to travel back to a single origin. The whole design collapses to one question: how do you keep the cache-hit ratio high and the origin protected, at a scale where one viral object can be requested millions of times a minute?
1. Requirements
A CDN is infrastructure rather than a customer-facing product: customers configure an origin and edge behavior, and the network has to serve enormous read fan-out while staying invisible when it works and predictable when it fails.
Functional requirements
Non-functional requirements
2. Capacity estimation
The numbers below exist to justify one design choice repeatedly: everything is built to keep as many requests as possible from ever leaving the edge.
| Metric | Estimate | Reasoning |
|---|---|---|
| Total edge requests / day | ~180 billion | 500M+ end users across all customer sites/apps, averaging ~350 requests/day/user (page assets, API calls, video segments). |
| Requests/sec, avg vs peak | ~2.1M avg, ~8M peak | 180B ÷ 86,400s ≈ 2.08M req/s average; live events and flash sales push peak to 3-4x average. |
| Cache hit ratio → origin reduction | 95% hit ratio | At 95%, only ~9B requests/day (~104K req/s avg) ever reach an origin shield — roughly a 20x reduction in origin-facing load. |
| Storage per edge node | 2-8 TB NVMe per server | Object popularity follows a Zipfian curve; 1-2TB of SSD/NVMe holds the working set that drives most hits, with a few GB of RAM caching the hottest tier. |
| PoPs / servers per PoP | ~300 PoPs, 20-200 servers each | Tier-1 metro PoPs run hundreds of servers; smaller regional PoPs run a handful — tens of thousands of edge servers globally in aggregate. |
| Egress bandwidth | ~150-250 Tbps aggregate peak | 180B requests/day at ~150KB average response implies ~27PB/day egress; peaks concentrate hard around live events and large releases. |
3. High-level design
Seven moving pieces cover the whole system: a client, a routing layer, the edge PoP itself (split into cache and compute), an origin shield, the origin, and a control plane that pushes configuration and purge events out to every PoP.
What each box owns
GeoDNS / Anycast router
Decides which PoP a given client talks to. GeoDNS resolves based on the resolver's location; anycast lets many PoPs advertise the same IP over BGP so normal internet routing sends the packet to the topologically nearest one. Either way, this layer owns health-aware routing: a PoP failing health checks is pulled out of rotation within seconds.
Edge PoP (cache + compute)
The workhorse. The edge cache (Varnish- or Nginx-style reverse proxy) owns TLS termination, cache lookups, and serving hits directly off local disk/RAM. Edge compute owns request/response transformation — header rewriting, A/B routing, small bits of logic — without a round trip to origin.
Origin shield
A thin caching tier that sits between hundreds of edge PoPs and one origin. It owns request coalescing: when 50 PoPs miss on the same object at once, the shield sends origin exactly one request and fans the response back out, instead of the origin seeing 50.
Purge / control plane
Owns the purge API, per-customer edge configuration (routing rules, cache-key overrides, TLS certs), and PoP health/capacity bookkeeping. It publishes changes rather than pushing them synchronously, so it stays available even when individual PoPs are slow to apply an update.
Config store
A small, low-latency store (etcd- or Redis-style) holding the current desired state — origin mappings, TLS certs, routing rules — that every PoP and the control plane read from. It is deliberately not the system of record for cached bytes, only for metadata about the system.
4. Architecture & deployment
Zoomed out, the network is a set of independent regions, each running many PoPs, all feeding through a shared origin-shield tier before anything reaches the customer's real infrastructure.
| Decision | Choice | Trade-off |
|---|---|---|
| Client-to-PoP routing | BGP anycast for the core IP, GeoDNS as a fallback/complement | Anycast gives near-instant failover (routing just converges) but less precise geo-control; GeoDNS is more precise but depends on resolver behavior and TTLs. |
| Origin shield | Yes, one shield tier per region per origin | Adds one hop of latency on a miss, but turns an N-PoP-to-1-origin fan-in into a handful of shield-to-origin connections — the single biggest origin-protection lever available. |
| Cache key design | Normalized path + sorted query params + explicit Vary allowlist | Coarser keys mean higher hit ratio but risk serving the wrong variant; finer keys (raw querystring, all headers) are safe but fragment the cache into near-zero hit ratio. |
| Purge propagation | Pub/sub fan-out (Kafka-style) to all PoPs, not synchronous RPC | Purge "completes" only once every PoP acks, which is eventually consistent for a short window — but a synchronous approach would make the API only as available as the slowest PoP. |
| Edge cache shard placement | Consistent hashing of the cache key across edge servers within a PoP | Keeps hot objects from landing on one disk and avoids full-cache reshuffles when a server joins or leaves, at the cost of extra hops within the PoP on a hash-ring change. |
5. Sequence flows
Two flows explain almost everything a CDN does: what happens on a cache miss, and what happens when someone asks the network to forget what it cached.
Cache miss through the origin shield
The shield exists precisely for step 3-4: every other PoP that also missed on this object during the same window is coalesced into the same single upstream request, so the origin sees one GET instead of hundreds. Steps 7-8 write the response into two caches on the way back, which is why the second request from anywhere else in that region is now a hit.
Cache invalidation / purge propagation
Purge is a broadcast, not an RPC: the control plane's job ends at step 2, and "complete" in step 7 means every PoP has acked the event, not that every in-flight request instantly saw new content. Step 8 is the important edge case — a request that started 50ms before the purge landed on its PoP is allowed to finish against the old object rather than hang, which keeps purge cheap without sacrificing correctness for more than a few seconds.
6. Control-plane data model
This is not a business schema — a CDN doesn't model users or orders, it models itself: which origins exist, which edge nodes are alive, and what's currently cached under which key. Cached bytes themselves live in local disk/object stores on each edge node, not in this metadata layer.
Key modeling decisions
| Store | SQL (e.g. Postgres) | NoSQL / KV (e.g. etcd, Redis) |
|---|---|---|
| origins, TLS certs | Good fit — low write volume, needs strong consistency and joins for billing/customer views. | Overkill; would need application-level consistency for something inherently relational. |
| edge_nodes health/heartbeat | Poor fit — heartbeats every few seconds from tens of thousands of nodes would thrash a relational table. | Good fit — TTL-based keys expire automatically, and reads are simple point lookups. |
| cache_keys / purge state | Workable at moderate scale, but write-heavy purge bursts can contend with the primary. | Good fit for the hot path (Redis for recent state); a log-structured store (Kafka) is better for the purge event stream itself. |
7. Deep dive: interview questions
How do you design the cache key so unrelated requests don't collide or fragment the cache?
The key has to be exactly as specific as it needs to be and no more. Start from the normalized path plus a sorted, allowlisted subset of query parameters — drop tracking params like utm_source entirely, since including them turns one cacheable URL into an unbounded number of cache entries. Then apply an explicit Vary allowlist (e.g. Accept-Encoding, sometimes Accept-Language) instead of trusting whatever the origin sends, because an unconstrained Vary on something like Cookie will fragment the cache down to a near-zero hit ratio. Session tokens and auth headers should never enter the key for genuinely cacheable content — if a response truly varies per user, it probably shouldn't be cached at the edge at all.
function cacheKey(req) {
const path = normalizePath(req.path);
const params = allowlistParams(req.query, ["v", "size"]).sort();
const vary = ["accept-encoding"].map(h => req.headers[h] || "");
return sha256(`${path}?${params.join("&")}|${vary.join("|")}`);
}Stale-while-revalidate vs a strict TTL — which do you pick and why?
Strict TTL means every expiry forces a synchronous origin round trip on the next request, which is exactly the thundering-herd risk you're trying to avoid for popular objects. stale-while-revalidate lets the edge serve the expired copy immediately while it revalidates in the background, so the client never waits on origin latency for an object that's a few seconds stale. Pair it with stale-if-error so a flaky origin degrades to "slightly old content" instead of an outage. The trade-off is consistency: for content that must be exactly current (a live price, a one-time signed URL) you accept the origin round trip; for nearly everything else, a short max-age plus a generous stale-while-revalidate window wins on both latency and origin protection.
Cache-Control: max-age=60, stale-while-revalidate=300, stale-if-error=3600A cache entry for a viral object expires and 50,000 concurrent requests hit at once — what happens?
Without protection, all 50,000 requests miss simultaneously and every one of them fires off toward the origin shield, and if the shield doesn't dedup, toward the origin — a self-inflicted DDoS at the exact moment the object is most popular. The fix is request coalescing (also called single-flighting): the first request to miss acquires a short-lived lock on that cache key, the other 49,999 requests are held (not queued to origin) and are released with the same response the moment the first one completes. This collapses N concurrent origin fetches into one, at the cost of the other 49,999 requests waiting slightly longer than they would have if they'd each fetched independently — a trade worth making every time.
Purge says "complete" in 2 seconds — what's actually true at that point, and what would make it slower?
"Complete" means the control plane has received an ack from every PoP subscribed to the purge topic, which is a strong statement about delivery but not an instant one about visibility — a request already in flight on some edge server when the purge event lands is typically allowed to finish against the old object rather than being torn down mid-response. What makes it slower: a straggler PoP with backlog on its subscriber, a network partition to one region requiring retry, or choosing a "hard purge" (evict immediately, next request always misses) over a "soft purge" (mark stale, serve stale-while-revalidate) — hard purge is simpler to reason about but reintroduces a small thundering-herd risk on the very next request for a popular object.
Why put an origin shield in front of the origin instead of letting every edge PoP hit it directly?
Without a shield, origin load scales with the number of edge PoPs and their per-PoP miss rate, not with actual content popularity — 300 PoPs each independently missing on the same freshly-purged object turns into 300 simultaneous origin requests for one logical fetch. The shield sits as a single (or per-region) hop that caches the same object with a longer TTL than the edge, and coalesces concurrent misses from every PoP behind it into one upstream request. Sizing it matters: too few shield nodes and it becomes its own bottleneck under a genuine spike; the usual approach is one small shield cluster per origin per region, scaled to a fraction of that region's edge fleet since it only ever sees the miss traffic, not the full request volume.
Post a Comment
Add