System design deep dive · HLD
Design a Load Balancer
Distribute millions of connections across a fleet of backends, detect failures in seconds, and never become the single point of failure yourself — the HAProxy / Envoy / AWS NLB problem, end to end.
1. Requirements
Before any box-and-arrow diagram: what must this system guarantee, and what are we explicitly not building?
Functional requirements
Non-functional requirements
2. Capacity planning
Rough numbers for a large multi-service platform — enough to size the LB tier and justify the health-check interval.
| Metric | Estimate | Note |
|---|---|---|
| Peak concurrent connections | ~5,000,000 | Kept-alive HTTP/2 and long-poll/WebSocket connections across all services |
| Peak requests/sec | ~900,000 RPS | Blended across L7 virtual hosts |
| Health-check overhead | ~18,000 pings/sec | 3,000 backends × 6 LB instances probing every 2s — under 2% of request volume |
| Backend instances per pool | 200 – 3,000 | Autoscaled; largest pools are stateless web/API tiers |
| LB throughput per instance | ~100,000 RPS / ~150,000 conns | Typical sustained figure for a tuned HAProxy/Envoy instance on modern hardware |
| LB instances needed | ~10 active (12 provisioned) | 900K RPS ÷ 100K per instance, plus N+2 redundancy buffer |
3. High-level design
Seven cooperating pieces: entry routing, an L4 tier, an L7 tier, the backend pool, a health checker, and a control plane that pushes configuration to all of them.
What each box owns
L4 load balancer tier
Operates at the transport layer — forwards TCP/UDP packets based on IP/port without inspecting payload. Extremely fast (kernel-level or DPDK-accelerated) and protocol-agnostic. This is the outermost tier: an AWS Network Load Balancer or a Linux IPVS cluster fronted by ECMP routing.
L7 load balancer tier
Terminates the connection, parses HTTP/gRPC, and routes on host, path, headers, or cookies. This is where content-based routing, retries, and session affinity live — think Envoy, nginx, or an AWS Application Load Balancer sitting behind the L4 tier.
Health checker
Runs active probes (HTTP/TCP pings on an interval) against every backend and also watches passive signals from live traffic (5xx rate, timeout rate). Publishes a health verdict per backend that the routing layer consumes before every decision.
Control plane / config store
Holds the source of truth for backend pool membership, routing rules, and algorithm settings (etcd, Consul, or a cloud target-group API). Pushes updates to every LB instance so they converge without a restart.
4. Deployment architecture
A single LB instance is a liability. The real design spreads redundant LB instances across availability zones and makes the LB tier itself horizontally scalable and self-healing.
| Decision | Choice | Why |
|---|---|---|
| L4 vs L7 placement | L4 in front of L7 | L4 absorbs raw connection volume and DDoS-scale packet rates cheaply; only traffic that survives reaches the more expensive L7 parsing tier. |
| Default algorithm | Least-connections (L7), consistent hashing for cache-affinity pools | Least-connections adapts to uneven request cost; consistent hashing keeps cache hit rates high when pool size changes. |
| Health-check strategy | Active + passive, combined | Active probes catch a dead process fast even with zero live traffic; passive signals catch slow degradation active probes miss. |
| LB-tier HA mechanism | ECMP + VRRP/keepalived (self-hosted) or cloud-native NLB (managed) | Removes the LB itself as a SPOF: a failed instance's VIP or ECMP route is withdrawn within seconds, no single box carries all traffic. |
5. Sequence diagrams
Two flows that define correctness: how one request picks a backend, and how a failing backend gets pulled out of rotation (and later rejoins).
Sequence 1 — L7 request distribution
The LB never guesses at backend health per-request — it consults the routing state that the health checker keeps fresh, filters to the healthy set, then applies the configured algorithm. Steps 3–5 typically run against an in-memory, per-instance cache of health state, not a network round trip, keeping this whole decision well under a millisecond.
Sequence 2 — Active health check failure and recovery
Consecutive-failure thresholds (not single-probe) avoid flapping a backend out over one transient blip. Recovery is intentionally asymmetric — fewer successes are required to rejoin than failures were required to leave — because rejoining slowly costs capacity, while leaving slowly costs correctness.
6. Control-plane data model
A load balancer has no "business" schema — what it persists is its own operating state: which backends exist, whether each is healthy, and which routing rule sends traffic where.
Key modeling decisions
| Store | Good fit when | Trade-off |
|---|---|---|
| etcd / Consul (KV + watch) | Every LB instance needs sub-second, push-based updates to converge | Weaker query flexibility than SQL; fine here since lookups are by key (pool_id, backend_id) |
| Relational DB (Postgres) | An operator-facing control UI needs ad-hoc queries, joins, audit history | Adds a poll-or-invalidate step before changes reach the hot routing path — extra latency to converge |
| Hybrid (common in practice) | Postgres as source of truth + etcd/xDS as the pushed, denormalized runtime view | Two systems to keep in sync, but gives both queryability and fast propagation |
7. Interview deep dive
L4 vs L7 — when do you actually need both?
L4 forwards packets by IP/port with no payload inspection: minimal CPU cost, protocol-agnostic (works for raw TCP, UDP, even non-HTTP protocols), and can sustain enormous packet rates because it never terminates the connection. Its ceiling is that it cannot make content-aware decisions — it doesn't know "/checkout" from "/health".
L7 terminates the TCP connection, parses the application protocol, and can route on host, path, method, or headers, plus do retries and circuit breaking. The cost is real: TLS termination, HTTP parsing, and buffering all consume CPU and add tens of microseconds to milliseconds of latency per hop.
Most large deployments use both: an L4 tier (AWS NLB, IPVS) absorbs raw connection volume and soaks up volumetric attack traffic cheaply, then hands surviving traffic to an L7 tier (Envoy/ALB/nginx) for the routing decisions that actually need content awareness.
Round robin vs least-connections vs consistent hashing — pick one for a scenario
Round robin fits a pool of near-identical, stateless backends handling uniformly cheap requests — e.g., a fleet of static-content edge nodes. It's simple and requires no shared state, but it ignores that some requests (a report export) may take 100x longer than others (a health ping), so it can overload a backend that got unlucky with a run of heavy requests.
Least-connections fits exactly that uneven-cost scenario — an API gateway mixing cheap reads and expensive writes. It routes to whichever backend currently has the fewest in-flight requests, self-correcting for request-cost variance, at the price of needing shared or per-instance connection counters.
Consistent hashing fits a caching layer or sharded service where you want the same client (or key) to keep landing on the same backend — e.g., a video-transcoding cache where re-hitting the same node avoids re-fetching source assets. Adding or removing one backend only remaps ~1/N of the keyspace instead of reshuffling everything, unlike naive modulo hashing.
function pickBackend(key, ring):
hash = consistentHash(key) // e.g. SHA-1(key)
node = ring.ceilingEntry(hash) // first node clockwise of hash
if node is null:
node = ring.firstEntry() // wrap around the ring
if not isHealthy(node):
node = ring.nextHealthy(node) // skip unhealthy nodes clockwise
return node.backendConnection draining during a rolling deploy
When a backend is about to be replaced, the control plane marks it "draining" instead of instantly removing it: the LB stops sending new connections to it immediately, but existing in-flight requests are allowed to finish, bounded by a drain timeout (commonly 30–300s depending on request length). Only after the timeout or when connection count hits zero is the instance actually terminated.
Skipping this step is the classic cause of deploy-time error spikes — killing an instance mid-request forces the client to retry (if idempotent) or simply fail. Draining decouples "stop giving me new work" from "shut down now."
How do you load-balance the load balancer itself?
The LB tier can't have a single instance in front of it, or that instance becomes the new SPOF. Three common patterns solve this without another LB in front:
- DNS/anycast: the same IP is announced from multiple physical locations; routers deliver each client to the topologically nearest, healthy instance.
- ECMP: upstream routers hash flows across several LB instances that all answer for the same virtual IP, spreading load without any single instance being "the" entry point.
- VRRP/keepalived: a floating virtual IP is owned by one active instance; on failure, a standby detects the missed heartbeat and takes over the VIP within a few seconds.
Cloud-managed options (AWS NLB, GCP's network load balancer) implement this pattern internally so the team never runs keepalived themselves — a strong default unless there's a reason to self-host.
The hot-backend problem: one instance is faster than the rest
If backends are heterogeneous (mixed instance types, one just finished a GC pause and is catching up, or one sits on faster hardware after a migration), naive round robin sends it the same share of traffic as a slower peer — either underutilizing the fast one or, worse, a "thundering herd" where least-connections floods the fastest backend the instant it looks least-loaded, then oscillates.
Fixes: weighted algorithms (assign capacity weights so a 2x backend gets 2x share), outlier detection (temporarily reduce traffic to instances with elevated latency/error rate, common in Envoy), and slow-start (a newly added or just-recovered backend receives ramped-up, not full, traffic for its first N seconds so it doesn't get slammed while still warming caches/JIT).
8. Summary
A load balancer is really two systems in one: a fast-path data plane that forwards packets/requests with minimal latency, and a slower control plane that keeps the data plane's view of "who is healthy" and "how to route" continuously up to date — while making sure the load balancer's own redundancy is never in question.
Post a Comment
Add