Mini project / POC · infrastructure
Distributed API Rate Limiter: a complete Spring Cloud Gateway mini project.
Token bucket vs. sliding window algorithms, atomic Redis Lua scripting that eliminates the classic check-then-increment race condition, multi-dimensional limiting by API key and route, and an honest answer to "what happens when Redis is down."
Project overview and requirements
"Design a rate limiter" is one of the most asked system-design questions there is, and most candidates can describe an algorithm but stumble on the part that actually matters: making it correct when the limiter runs as multiple horizontally-scaled instances sharing one counter. This project builds that correctly, not just conceptually.
Functional requirements
429, a Retry-After header, and remaining-quota headers.Non-functional requirements
Technology stack and why each piece is there
| Technology | Role in this project | Why this one |
|---|---|---|
| Java 21 + Spring Boot 3.x | Runtime and framework for both services. | Same reasoning as the other three mini projects. |
| Spring Cloud Gateway | Hosts the rate-limit filter on every request's critical path. | A reactive, non-blocking gateway is the natural place to reject a request before it ever reaches a backend service. |
| Redis | The single shared counter store every Gateway instance reads and writes. | Sub-millisecond in-memory operations, and critically, server-side Lua scripting for atomic check-and-increment. |
| PostgreSQL | Policy Service's durable store for tiers, limits, and API keys. | Rate-limit policy is low-volume, relational, admin-managed data — a normal database, not Redis, is the right home for it. |
| Spring Security + JWT | Protects the Policy Service's admin endpoints. | Same JWT/RBAC pattern as the other mini projects — only ADMIN can change limits. |
Jump to a section
High-level architecture
The Gateway is stateless and horizontally scaled — that's the whole reason the counter can't live in its memory. Redis is the one thing every Gateway instance agrees on, which is what makes "5 requests per second, globally" actually mean globally instead of "5 requests per second, per instance."
Service breakdown
API Gateway — where the rate limiter actually lives
A Spring Cloud Gateway instance with a custom GlobalFilter that runs before every proxied request. It resolves the caller's API key and route, looks up the applicable policy (from a short-TTL local cache backed by Policy Service), and executes one atomic Redis Lua script to decide allow or reject.
- routes all proxied traffic to downstream backend services
- rejects over-limit requests with
429before they reach any backend - caches policies locally (Caffeine, short TTL) to avoid a Policy Service call per request
Policy Service — tiers, limits, and API keys
Owns the relational configuration data: which tier an API key belongs to, and what limit and window each tier/route combination gets. This data changes rarely (an admin action), so it's a normal CRUD service backed by Postgres, not something that needs to sit on the hot request path.
GET /policies— Gateway, fetches current policy set (cached locally after)POST /policies,PATCH /policies/{id}— ADMIN onlyPOST /api-keys— ADMIN, issues a new API key for a client at a given tier
Rate limiting algorithms compared
There isn't one universally correct algorithm — each makes a different trade-off between memory cost, accuracy, and whether it allows short bursts. This project implements token bucket by default but the Redis key design supports swapping in any of them per route.
| Algorithm | How it works | Trade-off |
|---|---|---|
| Fixed window counter | Count requests in a hard time window (e.g. 00:00–00:59), reset to zero at the boundary. | Simplest and cheapest, but allows up to 2x the limit in a burst spanning a window boundary (a full quota just before 00:59, and another full quota just after 01:00). |
| Sliding window log | Store the timestamp of every request in a sorted set; count entries within the trailing window. | Perfectly accurate, but memory grows with request volume — expensive at high throughput. |
| Sliding window counter | Weight the previous fixed window's count by how much of it overlaps the current sliding window. | Close approximation of the sliding log at fixed-window memory cost — the common production compromise. |
| Token bucket (used here) | A bucket holds up to N tokens, refills at a steady rate, each request consumes one token. | Allows a controlled burst up to the bucket size while still enforcing a steady-state average rate — matches real client behavior better than a hard window. |
The race condition, and how atomic Lua scripting fixes it
This is the part almost every "design a rate limiter" answer glosses over. Doing the token-bucket check and the token-bucket update as two separate Redis calls from the application is broken under concurrency — and it's the single most important correctness detail in this entire project.
-- token_bucket.lua
-- KEYS[1] = bucket key, ARGV[1] = capacity, ARGV[2] = refill rate/sec, ARGV[3] = now (epoch seconds)
local bucket = redis.call('HMGET', KEYS[1], 'tokens', 'last_refill')
local tokens = tonumber(bucket[1]) or tonumber(ARGV[1])
local lastRefill = tonumber(bucket[2]) or tonumber(ARGV[3])
local elapsed = tonumber(ARGV[3]) - lastRefill
local refill = math.min(tonumber(ARGV[1]), tokens + elapsed * tonumber(ARGV[2]))
if refill >= 1 then
redis.call('HMSET', KEYS[1], 'tokens', refill - 1, 'last_refill', ARGV[3])
redis.call('EXPIRE', KEYS[1], 3600)
return 1 -- allowed
else
redis.call('HMSET', KEYS[1], 'tokens', refill, 'last_refill', ARGV[3])
redis.call('EXPIRE', KEYS[1], 3600)
return 0 -- rejected
end// RateLimiterFilter.java
@Component
public class RateLimiterFilter implements GlobalFilter, Ordered {
private final ReactiveRedisTemplate redis;
private final RedisScript tokenBucketScript; // loads token_bucket.lua
@Override
public Mono filter(ServerWebExchange exchange, GatewayFilterChain chain) {
String apiKey = exchange.getRequest().getHeaders().getFirst("X-API-Key");
RateLimitPolicy policy = policyCache.forKeyAndRoute(apiKey, exchange.getRequest().getPath().value());
String bucketKey = "ratelimit:{" + apiKey + "}:" + policy.routeGroup();
return redis.execute(tokenBucketScript,
List.of(bucketKey),
List.of(String.valueOf(policy.capacity()), String.valueOf(policy.refillPerSecond()), String.valueOf(nowEpochSeconds())))
.single()
.flatMap(allowed -> allowed == 1L
? chain.filter(exchange)
: reject(exchange, policy));
}
} EVAL is running, no other command touches that key. That guarantee, not anything in the Java code, is what makes the check-and-decrement atomic.Multi-dimensional limiting: Redis key design
A real API needs more than one limit dimension — per API key, per route, sometimes per IP as a secondary defense. Each dimension is just a different Redis key built from the same script.
| Dimension | Key pattern | Purpose |
|---|---|---|
| Per API key + route | ratelimit:{apiKey}:route:/orders | The primary business limit — this client's plan allows N requests/sec to this endpoint. |
| Per API key, global | ratelimit:{apiKey}:global | An overall ceiling regardless of which routes the client hits. |
| Per IP | ratelimit:{clientIp}:ip | A coarse, identity-independent defense against abuse from an unauthenticated or spoofable source. |
A single request can be checked against more than one bucket (both the per-key and per-IP limits), and is rejected if any applicable bucket is empty — the same Lua script, called once per dimension that applies.
{apiKey} hash tag matters in Redis Cluster Redis Cluster shards keys across nodes by hashing the key, but a Lua script can only atomically touch keys that live on the same node. Wrapping the variable part of the key in {} (a hash tag) forces Redis to hash only that substring, guaranteeing every key for one API key lands on the same node — without it, the script would fail cross-slot in a clustered deployment.Security and role-based access control
Two different identities matter here: the API key being rate-limited (a client's traffic identity) and the JWT-authenticated admin managing policies (a human's identity). They're deliberately separate concerns.
| Role | Can do |
|---|---|
| API key (any tier) | Make requests through the Gateway; subject to its tier's limits. Not a JWT-authenticated identity. |
ROLE_ADMIN (JWT) | Create/modify rate-limit policies and issue new API keys via Policy Service. |
Request sequence and policy propagation
When Redis goes down: fail-open vs. fail-closed
The rate limiter has a single point of dependency on Redis, and what happens during a Redis outage is a real, consequential design decision — not an afterthought.
return redis.execute(tokenBucketScript, keys, args)
.single()
.timeout(Duration.ofMillis(50)) // fail fast, don't let Redis latency become request latency
.onErrorReturn(1L) // fail OPEN: treat any Redis error as "allowed"
.flatMap(allowed -> allowed == 1L ? chain.filter(exchange) : reject(exchange, policy));Project folder structure
api-rate-limiter-poc/ ├── api-gateway/ │ ├── src/main/java/com/jiquest/gateway/ │ │ ├── GatewayApplication.java │ │ ├── filter/RateLimiterFilter.java │ │ ├── filter/ResponseHeaderFilter.java // X-RateLimit-* headers │ │ ├── config/RedisConfig.java // loads token_bucket.lua as a RedisScript bean │ │ ├── policy/PolicyCache.java, PolicyClient.java │ │ └── policy/RateLimitPolicy.java │ ├── src/main/resources/application.yml │ ├── src/main/resources/scripts/token_bucket.lua │ ├── Dockerfile │ └── pom.xml │ ├── policy-service/ │ ├── src/main/java/com/jiquest/policy/ │ │ ├── PolicyServiceApplication.java │ │ ├── config/SecurityConfig.java │ │ ├── controller/PolicyController.java, ApiKeyController.java │ │ ├── service/PolicyService.java, ApiKeyService.java │ │ ├── repository/PolicyRepository.java, ApiKeyRepository.java │ │ └── entity/RateLimitPolicy.java, ApiKeyEntity.java │ ├── src/main/resources/application.yml │ ├── src/test/java/com/jiquest/gateway/ConcurrentLimitIT.java │ ├── Dockerfile │ └── pom.xml │ ├── docker-compose.yml ├── .env.example └── README.md
Local development setup
# docker-compose.yml (excerpt)
services:
redis:
image: redis:7-alpine
ports: ["6379:6379"]
postgres-policy:
image: postgres:16
environment: { POSTGRES_DB: policy_db, POSTGRES_PASSWORD: postgres }
ports: ["5433:5432"]
policy-service:
build: ./policy-service
depends_on: [postgres-policy]
ports: ["8081:8080"]
api-gateway:
build: ./api-gateway
depends_on: [redis, policy-service]
environment:
SPRING_REDIS_HOST: redis
POLICY_SERVICE_URL: http://policy-service:8080
ports: ["8080:8080"]Testing strategy: proving the limit actually holds
Like the payment project's concurrency test, the test that matters most here isn't a unit test of the Lua script's logic in isolation — it's firing real concurrent requests and proving the aggregate count never exceeds the configured limit.
@Testcontainers
class ConcurrentLimitIT {
@Container
static GenericContainer redis = new GenericContainer<>("redis:7-alpine").withExposedPorts(6379);
@Test
void fiftyConcurrentRequestsAgainstALimitOfTenAllowExactlyTen() throws InterruptedException {
ExecutorService pool = Executors.newFixedThreadPool(50);
CountDownLatch latch = new CountDownLatch(50);
AtomicInteger allowed = new AtomicInteger();
for (int i = 0; i < 50; i++) {
pool.submit(() -> {
if (rateLimiter.tryConsume("test-key", "route:/orders")) {
allowed.incrementAndGet();
}
latch.countDown();
});
}
latch.await(10, TimeUnit.SECONDS);
assertThat(allowed.get()).isEqualTo(10); // never 11, regardless of thread interleaving
}
}Observability
Path to production on AWS
| Local (this POC) | AWS equivalent |
|---|---|
| Single Redis container | Amazon ElastiCache for Redis, cluster mode enabled for scale (hash-tagged keys still required) |
| N Gateway containers behind Docker Compose | ECS Fargate or EKS, auto-scaled behind an ALB |
| Postgres container for policy_db | Amazon RDS PostgreSQL — low volume, doesn't need Multi-AZ as urgently as the payment project's ledger did |
Key design decisions and interview talking points
Why can't each Gateway instance just keep its own in-memory counter?
With an in-memory counter and 5 Gateway instances behind a load balancer, a client sending requests that get spread across all 5 would effectively get 5x the intended limit — each instance only sees and limits its own slice of traffic. A shared external store is what makes "N requests per second" mean N in total, not N per instance.
Why token bucket instead of the sliding window log, which is more accurate?
Sliding window log's accuracy comes from storing every request's timestamp, which costs memory proportional to request volume — expensive at real API scale. Token bucket needs only two numbers per key (current tokens, last refill time) regardless of traffic volume, and its burst-tolerance behavior actually matches how real clients behave (occasional bursts, not perfectly metronomic traffic) better than a hard window does.
What exactly goes wrong with a naive GET-then-SET implementation, concretely?
Two concurrent requests can both execute GET before either executes SET, so both read the same "1 token available" state and both proceed to allow the request — the second request's decrement overwrites, rather than compounds, the first's. The window for this race can be microseconds, but under real concurrent load it happens constantly, not rarely.
Why does the fix have to be a server-side Lua script instead of a client-side Redis transaction (MULTI/EXEC)?
Redis transactions queue commands and execute them atomically, but a transaction can't make a decision based on a value it just read within the same transaction — you'd still need to read the value first, outside the transaction, to decide whether to proceed, reopening the same race. A Lua script runs entirely server-side, so the read, the decision, and the write happen as one atomic unit with no gap for another client to interleave.
How do you rate-limit by both API key and IP without one dimension unfairly penalizing shared infrastructure?
The per-API-key limit is the primary, fair business limit tied to a specific client's plan; the per-IP limit is a coarser, secondary defense meant to catch abuse from unauthenticated or unidentified traffic, and is typically set much higher so it rarely affects legitimate traffic from behind a shared NAT or corporate proxy where many real users share one IP.
Why cache policies in the Gateway locally instead of calling Policy Service on every request?
Policy Service sits outside the hot path specifically so a slow or unavailable policy lookup doesn't become added latency (or a new failure mode) on every single proxied request. A short-TTL local cache means policy changes still propagate within seconds, which is more than fast enough for a limit change, while removing Policy Service from the critical path entirely.
What's the Redis Cluster hash-tag issue, and why would the Lua script break without it?
Redis Cluster shards keys across nodes based on a hash of the key, and a Lua script (or any multi-key command) can only run atomically across keys that resolve to the same node. Without a {apiKey} hash tag forcing colocation, two keys for the same client could land on different nodes, and Redis would refuse the script with a cross-slot error in cluster mode — it works by accident on a single-node Redis and breaks the moment you cluster it.
Why include a Retry-After header instead of just returning a bare 429?
A bare 429 tells a well-behaved client it was rejected but not when to safely retry, so it either retries immediately (making the problem worse) or backs off arbitrarily. Retry-After (and the X-RateLimit-Remaining/Reset headers) let a well-behaved client's own retry logic cooperate with the limiter instead of guessing.
How would clock skew across Gateway instances affect correctness, given the Lua script takes "now" as an argument?
Each Gateway instance passes its own local clock's epoch time into the script, so meaningful skew between instances could let a client get a slightly different effective refill rate depending on which instance handled which request. This is why the script's "now" parameter should ideally come from Redis's own TIME command instead of the caller's clock — a refinement worth making before this leaves POC stage.
Is rate limiting the same thing as backpressure or throttling?
They're related but distinct: rate limiting rejects excess requests outright once a threshold is crossed, a hard admission-control decision at the edge. Backpressure is a downstream system signaling it's overloaded so an upstream caller slows down or queues, and throttling often refers to deliberately slowing (not rejecting) requests to smooth load. This project implements rate limiting specifically — rejection, not queuing or slowing.
How would you test that the limiter is actually atomic, rather than just trusting the Lua script's logic?
A unit test of the script's branching logic proves the algorithm is correct in isolation, but says nothing about concurrent correctness; the real proof is firing many concurrent requests (as in the testing section above) against a real Redis instance and asserting the allowed count exactly matches the limit, which would fail immediately if the atomicity guarantee were somehow broken.
What happens if Redis itself becomes the bottleneck at very high request volume?
A single Redis node has a throughput ceiling; the standard mitigations are Redis Cluster (sharding keys, using the hash-tag technique above to keep each client's keys atomic-safe), or accepting a slightly weaker consistency model (a local-first counter with periodic async sync to Redis) that trades perfect global accuracy for much higher throughput — a real trade-off to make only once you've measured that Redis, not something else, is actually the bottleneck.
Why does an unrecognized API key get the strictest tier instead of being rejected outright or given a generous default?
Rejecting outright would break legitimate new clients mid-provisioning; a generous default would let a malformed or missing key accidentally bypass rate limiting entirely. Defaulting to the strictest anonymous tier fails toward safety in both directions — it degrades gracefully rather than either blocking valid traffic or exposing an unlimited loophole.
Why is Policy Service a separate service instead of just a config table the Gateway reads directly?
Separating it gives policy management its own access control, audit trail, and API independent of the Gateway's deploy lifecycle — an admin can safely change a limit without touching or redeploying the Gateway at all, and the same service could later serve a billing dashboard or usage-reporting feature reusing the same policy and API-key data.
If an interviewer asks for the single most important design decision in this project, what do you say?
Making the check-and-decrement operation atomic via a server-side Lua script rather than two separate client-side Redis calls — it's the one decision that determines whether the rate limiter is actually correct under concurrency or just looks correct in a single-threaded demo, which is exactly the gap between a textbook algorithm description and a production-grade implementation.
Related guides
Update these hrefs to your published Blogger post URLs once each page is live.
Post a Comment
Add