System design deep dive · HLD
Design an API Gateway: full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for authenticated routing and for rate limiting, and an entity-relationship diagram for routes, API keys, and policies - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
An API gateway is a single choke point that every request passes through, which means its design is dominated by one constraint: it must add negligible latency and never become the thing that goes down, even while enforcing auth, quotas, and routing on every single call.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide whether rate-limit counters can live in a single Redis instance, how large the connection pool to each backend needs to be, and how quickly route/policy changes must propagate to avoid stale enforcement.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Peak requests routed | A mid-size platform fronting ~40 backend services | ~200,000 requests/sec peak across all routes combined |
| Rate-limit checks | Every request checks a token bucket | 200,000 checks/sec - must be an in-memory/Redis counter operation, never a relational row-level lock |
| Distinct API keys / policies | ~500,000 active API keys, ~200 distinct route policies | Small enough that the full route table and policy set fit comfortably in each gateway node's local memory cache |
| Auth token validation | JWT signature verification is CPU-bound, not I/O-bound | ~200,000 verifications/sec spread across a horizontally scaled gateway fleet - stateless, no database round-trip per request |
| Config/policy change rate | Route and policy updates happen ~dozens of times/day, not per-request | Low write volume; the win is in propagation speed (push, not poll) rather than write throughput |
3. High-level design (HLD)
The HLD treats the gateway as a request pipeline of independent, ordered stages - auth, rate limit, route resolution, proxy - each backed by its own fast local cache, with a separate control plane responsible for pushing configuration changes to every node.
What each box owns
Auth stage
Validates the caller's identity - either a signed JWT (verified against a locally cached JWKS public key set, no network call per request) or an API key (looked up in a local in-memory cache refreshed from the control plane). A request that fails auth is rejected here, before it ever touches rate limiting or routing, saving downstream work.
Rate-limit stage
Checks the caller's token bucket for the specific route being called. The bucket state lives in a shared, fast counter store (Redis, using atomic INCR/Lua scripts) so that all gateway nodes enforce the same limit consistently for a given key, rather than each node keeping an independent (and therefore gameable) local counter.
Route resolver & service registry
Matches the request path/host against the cached route table to determine the target backend service, then resolves that service name to a live, healthy instance via the service registry (Consul/etcd/Kubernetes DNS) - never a hardcoded IP. This is what lets backend services scale up/down or move without any gateway configuration change.
Control plane / admin API
The single source of truth for routes, API keys, and policies, exposed via an internal admin API used by platform engineers (or a CI/CD pipeline) to publish changes. It never sits on the request hot path itself - instead it pushes invalidations/updates to every gateway node's local cache, which is what keeps per-request latency low while still letting configuration change in near real time.
4. Detailed architecture diagram
The architecture diagram shows the gateway deployed as a stateless, horizontally scaled fleet behind a load balancer, a shared rate-limit counter store, and a control plane that pushes config changes via pub/sub instead of every node polling a database.
| Decision | Choice | Reasoning |
|---|---|---|
| Rate-limit state | Shared Redis cluster, not per-node local counters | A client hitting different gateway nodes across requests must still be limited consistently; local-only counters would let a client multiply its effective quota by the number of nodes. |
| Route/policy distribution | Push (pub/sub) from control plane to local node cache | 200,000 req/sec cannot afford a database round-trip per request; pushing invalidations keeps reads local while still propagating changes in seconds, not the minutes a polling interval would need. |
| Per-backend isolation | Independent connection pool + circuit breaker per backend | A slow or failing "Payments" backend must not exhaust the shared thread/connection pool and start failing unrelated "Inventory" requests too. |
| Service discovery | Dynamic registry (etcd/Consul/K8s DNS), not static config | Backends scale, restart, and get rescheduled constantly; hardcoded IPs would require a gateway config change (and redeploy) for every backend topology change. |
5. Sequence diagrams for the two critical flows
Authenticated routing shows the full happy-path pipeline order; rate limiting shows exactly how a shared counter store makes throttling consistent across a horizontally scaled gateway fleet.
5.1 Authenticated request routing to a backend service
Steps 2-4 are all local, in-memory operations on the gateway node - no network call, no database query - which is exactly how the pipeline stays under a few milliseconds of added latency. Step 5 is the only external lookup in the fast path, and it's typically served from the registry's own local cache on the gateway node too, refreshed on a short interval or via watch/subscribe rather than queried synchronously per request in practice.
5.2 Rate limiting with a shared token bucket
Step 2 uses an atomic Lua script in Redis rather than a separate GET-then-SET, so two concurrent requests for the same key from different gateway nodes can never both read the same token count and both decide they're allowed - the check-and-decrement is a single atomic operation server-side. This is precisely why a shared counter store (not per-node local counters) is required in step 5-7: node B correctly denies the request because it consults the same shared state node A wrote to, even though B has never seen this client before.
6. Entity-relationship (ER) diagram and schema
The schema has to answer: how is an incoming path matched to exactly one route, how is an API key's identity and quota looked up in constant time, and how are rate-limit/auth policies attached to a route without duplicating them per route.
Key modeling decisions
/v1/orders/* covers an entire resource family instead of one row per exact endpoint, keeping the route table small enough to cache fully in memory.| Storage choice | Use when | Watch out for |
|---|---|---|
| Relational (Postgres) as the control-plane source of truth | Strong consistency matters for who-can-call-what, and the write volume is low (config changes, not requests). | Never queried on the request hot path directly - always read from each gateway node's local cache. |
| In-memory (Redis) for rate-limit counters | Access pattern is atomic increment/check at extreme request volume, shared across a fleet. | Counter state is intentionally ephemeral; a Redis failover briefly resets in-flight windows, which is an accepted trade-off for the speed gained. |
7. Deep dives interviewers actually probe
Token bucket vs sliding window vs fixed window - which rate-limiting algorithm and why?
A fixed window (e.g. "100 requests per calendar minute") is simple but allows a burst of 200 requests right across a window boundary (100 at 0:59, 100 at 1:00). A sliding-window log is precise but requires storing a timestamp per request, which is memory-expensive at scale. A token bucket is the common middle ground: it allows controlled bursting up to the bucket size while enforcing a steady average refill rate, with O(1) state per key (just a count and a last-refill timestamp) - the reasoning above uses it for exactly that memory-efficiency and burst-tolerance balance.
-- Redis Lua script: atomic token-bucket check (simplified)
local tokens = tonumber(redis.call('GET', KEYS[1]) or capacity)
local elapsed = now - last_refill
tokens = math.min(capacity, tokens + elapsed * refill_rate)
if tokens < 1 then return 0 end
redis.call('SET', KEYS[1], tokens - 1)
return 1What happens when a backend service is slow or down?
Each backend gets its own connection pool and circuit breaker. Once error rates or latency for a specific backend cross a threshold, the circuit opens: the gateway immediately fails fast (returns 503) for that backend's routes instead of piling up slow, doomed requests and exhausting shared resources - and periodically sends a small number of trial requests to detect recovery (half-open state) before fully reopening the circuit.
How does the gateway avoid becoming a single point of failure itself?
The gateway is deployed as a stateless, horizontally scaled fleet behind a load balancer across multiple availability zones - any single node can fail without dropping traffic. The genuinely stateful dependencies it leans on (rate-limit counters, service registry) are themselves deployed as clustered, replicated systems, and the gateway is built to degrade gracefully if one of them is briefly unavailable - e.g., falling back to a slightly stale local route cache rather than rejecting all traffic if the control plane is unreachable.
JWT vs opaque API key - why support both, and how does verification differ?
A JWT is self-contained and verifiable locally (signature check against a cached public key) with zero network calls, ideal for high-volume, short-lived user sessions, but it can't be instantly revoked before its expiry without an extra denylist check. An opaque API key requires a lookup (typically against a local cache refreshed from the control plane) but can be revoked instantly by simply removing it, which is why long-lived machine-to-machine credentials are usually modeled as API keys rather than JWTs.
What is the single biggest bottleneck as this scales 10x?
Not the gateway compute itself - it's stateless and scales linearly by adding nodes. The real bottleneck becomes the shared rate-limit counter store: at 2 million checks/sec, a single Redis cluster's network and CPU capacity for atomic Lua-script executions becomes the constraint. The fix is sharding the counter keyspace across multiple Redis clusters (e.g. by hash of api_key) rather than scaling a single cluster indefinitely, accepting slightly more complex client-side routing logic in exchange.
Post a Comment
Add