System Design Interview · HLD Series
Design a Real-Time Leaderboard (Gaming)
Hundreds of millions of players, thousands of score updates per second, and a rank query that must answer in single-digit milliseconds. This is the interview question where "just use a database" quietly falls apart — and where recognizing that fact is the whole point.
1. Requirements
Before touching architecture, pin down exactly what "leaderboard" means here — a single global list behaves very differently from per-region, per-game-mode, seasonal boards that reset every few weeks.
Functional
Non-Functional
2. Capacity Estimation
The numbers below drive two decisions later: whether a single Redis instance can hold everything (it can't), and how many shards we need.
| Dimension | Estimate | Why it matters |
|---|---|---|
| Daily active players | ~250 million | Sets the size of the "player universe" per leaderboard season. |
| Peak score updates / sec | ~40,000/s (tournament peak, ~8x average) | Drives write throughput per shard and shard count. |
| Bytes per ZSET entry | ~80–100 bytes (member id + score + skip-list pointers) | 250M entries × ~90B ≈ ~22–24 GB just for one global board. |
| Shards needed | ~24 shards at ~10GB usable RAM each (with headroom + replicas) | Shard key: hash(player_id) for the write path; a secondary global-rank aggregation layer stitches shards together for reads. |
| Top-K query rate | ~200,000 reads/sec (leaderboard screen views) | Cacheable: top-100 changes slowly relative to view volume — cache with a short TTL. |
| Rank-of-player query rate | ~15,000 reads/sec ("what's my rank" after a match) | Not cacheable per-player at scale; must hit the shard directly, cheaply, via O(log N) skip-list rank. |
3. High-Level Design
Six moving parts: a write path that validates and stores, a read path that ranks and serves, and a durability layer underneath both so a node crash doesn't erase a tournament's final minutes.
What each box owns
Score-Update Service
Terminates the write request from the game server (never directly from an untrusted client), stamps it idempotent per match id so a retry doesn't double-count, and forwards it to validation before it ever touches the ZSET.
Anti-Cheat / Score-Validation
Applies bounds checks (a single match cannot plausibly add 500,000 points), rate limits (a player cannot submit 200 score deltas in one second), and cross-checks against server-authoritative match telemetry rather than trusting the client's claimed delta.
Redis ZSET Cluster
The source of truth for "current standings." Each shard is a Redis sorted set keyed by player id with score as the sort key, backed by a skip list internally so insert, update, and rank are all O(log N). Sharded so no single node has to hold all 250M players.
Season / Archive Store
A durable, queryable store (Postgres or a column store) that receives a snapshot of final standings when a season closes. Not on the hot path — it exists so "what was my rank last season" doesn't require keeping every season live in Redis forever.
4. Deeper Architecture: Sharding & Cross-Shard Rank
The interesting engineering is entirely in how we shard a data structure whose whole purpose is a global total order, and what we give up to make that shardable.
| Decision | Choice | Trade-off |
|---|---|---|
| Data structure | Redis ZSET (skip list under the hood) | O(log N) insert/update/rank vs. an SQL table where ORDER BY + OFFSET degrades linearly and an indexed UPDATE re-balances a B-tree on every score change. |
| Sharding key | hash(player_id) across N shards | Writes distribute evenly and a player's own rank query hits exactly one shard — but a truly exact global top-K or global rank now requires merging across every shard. |
| Rank computation | Real-time ZREVRANK per shard + periodic global merge | A player's local-shard rank is exact and instant; their global rank is an approximation reconciled every few seconds by the aggregator, not recomputed from scratch on every request. |
| Persistence | AOF (append-only file) + periodic RDB snapshot | AOF gives near-zero data loss on crash at the cost of write overhead; RDB gives fast full-cluster restore at the cost of losing the last few seconds since the last snapshot. Running both hedges each weakness. |
5. Sequence Flows
Two flows worth tracing step by step: submitting a score, and the much trickier problem of answering "what's my global rank" when the answer lives across many independent shards.
Sequence 1: Score Update
The write itself (step 4) is a single ZADD, which is O(log N) regardless of how many players are already in the set. Anti-cheat runs synchronously and cheaply (a bounds/rate check, not a full replay), so it doesn't add meaningful latency. The Kafka publish in step 7 is fire-and-forget: it feeds real-time UI updates (a live "you moved up 3 spots" toast) without gating the player's own request on a downstream consumer.
Sequence 2: Rank-of-Player Query Across Shards
Step 5–6 is the crux of sharded leaderboards: to know a player's true global rank, you need to know how many players on every other shard scored higher — there's no way around touching every shard for an exact answer. In practice we fan out ZCOUNT(score, +inf) calls in parallel (they're cheap, O(log N) each) and sum the results. Because other shards may have shifted slightly between the fan-out and the response, the number returned is a tightly-bounded estimate, not a linearizable exact rank — and the response says so explicitly rather than pretending otherwise.
6. Data Model
Unusually for this series, the "database" here largely is the live in-memory ZSET — there's no separate control-plane database sitting in front of it. The entities below describe the system's actual persisted state: the scores themselves, the shard topology, and the season boundaries that govern resets.
Key modeling decisions
| Approach | Redis ZSET (in-memory) | Persistent DB with indexed rank column |
|---|---|---|
| Insert / update | O(log N), skip list, sub-millisecond | O(log N) index update but with disk I/O and lock contention under high concurrency |
| Rank query | O(log N) native ZREVRANK | Requires a window function or COUNT(*) WHERE score > x, expensive at scale without careful indexing |
| Durability | Needs AOF/RDB; in-memory by default | Durable by default (WAL/redo log) |
| Best fit | Live, hot, frequently-changing standings | Cold archive of a closed season for analytics/history queries |
7. Interview Deep Dive
Why a sorted set instead of "just ORDER BY score in SQL"?
A relational table with an index on score can answer "top 10" reasonably well, but two things break down at leaderboard scale. First, rank — "what position is player X in" — requires counting how many rows have a higher score, which is a COUNT(*) scan unless you maintain a separate materialized rank column, and that column has to shift for potentially millions of rows every time someone crosses another player's score. Second, write amplification: every score UPDATE has to re-balance a B-tree index under concurrent load from tens of thousands of writers per second.
A Redis sorted set is backed by a skip list plus a hash table. The skip list keeps members ordered by score with express lanes that let both insertion and rank lookup skip large sections of the list, giving O(log N) for ZADD, ZRANK, ZREVRANK, and ZRANGE alike — and unlike a SQL table, no separate "rank column" needs to be maintained; rank is a structural property of the skip list itself, always current.
ZADD leaderboard:global:s7 98240 "player:novablade"
ZREVRANK leaderboard:global:s7 "player:novablade"
> (integer) 1 # 0-indexed, so this player is rank 2
ZREVRANGE leaderboard:global:s7 0 9 WITHSCORES
> 1) "player:novablade" 2) "98240"
> 3) "player:kestrel99" 4) "97910"
> ...
You sharded the ZSET. How do you get an exact global rank now?
You don't, exactly — and saying so plainly is the right answer. A single ZSET gives an exact global rank for free because it's one ordered structure. Splitting it across N shards for write scalability means no single shard knows the global order; a player's rank within their own shard is exact and instant, but their true global rank requires knowing how many higher scores exist on every other shard too.
The practical fix is fan-out plus bound: query ZCOUNT(score, +inf) on every other shard in parallel (cheap, O(log N) each), sum the counts, and add the player's local rank. The result is accurate to within whatever drift accumulated across shards during the fan-out window — typically single-digit-millisecond staleness. For top-K (not rank-of-one-player), the same idea runs in reverse: merge each shard's local top-K candidates and re-sort, which is exact for "who's in the top 100" as long as no shard's 101st-ranked player could have snuck into the merged top 100 — which holds if K is modest relative to shard size.
How do you reset a season without downtime, and without losing history?
Never delete-then-rebuild in place. The sequence is: (1) at season end, walk every shard and stream its full ZSET contents (ZRANGE in batches) into the season archive store, tagged with the closing season_id; (2) once every shard confirms its snapshot landed, atomically swap each shard's live key to a fresh, empty key (Redis RENAME is O(1) and near-instant) rather than deleting members one by one; (3) update the seasons table to mark the old season archived and the new one active. Because the swap is a key rename rather than a bulk delete, there's no window where the leaderboard is empty or half-migrated, and a crash mid-archive just means retrying the snapshot step against the still-live old key.
One game mode is 10x more popular than the rest — how do you avoid a hot shard?
Sharding purely by hash(player_id) assumes uniform traffic per player, which breaks the moment a viral game mode or a live tournament concentrates both writes and rank-queries onto whichever shards happen to hold those players. Two complementary mitigations: first, give especially hot boards (a global tournament leaderboard, a launch-day event) their own dedicated, separately-scaled ZSET keys rather than letting them share the general player-score shards — isolating blast radius. Second, for read amplification specifically, put read replicas behind the hot shard's primary and route ZREVRANK/ZREVRANGE reads to replicas, keeping the primary free for writes. If a single mode's write volume alone saturates one shard, sub-shard it further by a secondary key (e.g., hash(player_id) combined with a mode-specific salt) and merge across those sub-shards the same way we merge across regular shards.
How do you stop a client from just submitting a fabricated score?
The score-update endpoint should never trust a client-supplied absolute score or an unchecked delta. Three layers: first, the score delta should be computed and signed server-side by the authoritative game server at match end, not read from the client's request body — the client reports "match M ended," the game server (which already knows the real outcome) is the one asserting the point value. Second, bounds and rate checks catch what slips through: a delta far outside the plausible range for one match, or a player submitting updates far faster than a match could legitimately conclude, gets flagged or held for review rather than applied immediately. Third, idempotency keyed on match_id prevents a replayed or duplicated request from being double-counted, which is a more common "cheat" vector in practice than outright fabrication.
8. Summary
A real-time leaderboard looks deceptively simple — sort some numbers, show the top ones — until scale forces a genuine trade-off: an exact global total order and horizontal sharding are in direct tension, and production systems resolve that tension by shipping a tightly-bounded approximation rather than pretending both can be had for free.
Post a Comment
Add