System design deep dive · HLD
Design a Distributed Message Queue (Kafka-like event log): full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level component diagram, a deeper deployment architecture diagram with partitions and replicas spread across brokers, sequence diagrams for the produce-with-acks and consume-with-offset-commit paths, and an entity-relationship diagram for the cluster metadata model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
A strong system design answer starts by pinning down scope out loud. For a Kafka-like distributed message queue, that means separating what the system must guarantee from how fast it must go, and being explicit that ordering and delivery guarantees are scoped to a partition, not the whole topic.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide everything downstream: how many partitions the cluster needs at minimum, how many brokers are required to absorb replicated write I/O, and how much disk the retention window actually costs.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Average ingest | 2,000,000 messages/sec across all topics, ~1 KB average message size | 2,000,000 × 1KB ≈ 2 GB/sec average write throughput |
| Peak ingest | ~3× average during traffic spikes | ≈ 6 GB/sec peak write throughput |
| Minimum partition count | a single partition tops out around 15 MB/sec of sequential disk write on typical disks | 2 GB/sec ÷ 15 MB/sec ≈ 134 partitions minimum, provisioned generously to ≈ 5,000 partitions across ≈ 800 topics for consumer parallelism headroom |
| Replication factor | RF = 3 (1 leader + 2 followers) for durability | write I/O triples cluster-wide: 2 GB/sec logical → ≈ 6 GB/sec of physical disk writes |
| Brokers needed | each broker sustains ≈ 250 MB/sec durable sequential write across its disks | 6 GB/sec ÷ 250 MB/sec ≈ 24 brokers minimum, provisioned to ≈ 150 brokers across 3 AZs for headroom and rack spread |
| Retention storage | 7-day retention window, RF = 3 | 2 GB/sec × 604,800 sec × 3 ≈ 3.6 PB of hot storage cluster-wide |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flow between them, without committing yet to specific broker counts, racks, or regions - that level of detail belongs in the architecture diagram in the next section.
What each box owns
Producers & the partitioner
Producers batch and compress messages client-side, then a partitioner hashes each message's key to pick a partition (or round-robins for keyless messages). The key choice determines both throughput distribution (a hot key skews load to one partition) and ordering guarantees (only messages sharing a key are guaranteed relative order).
Broker cluster: partition leaders and follower replicas
Every partition has exactly one leader broker at a time; all produces and consumes for that partition go through the leader. Follower brokers replicate the leader's log and can be promoted to leader instantly if the leader dies, which is why replicas are always placed on different brokers/racks than the leader.
Controller / metadata layer
A small quorum (a Raft-based controller, e.g. KRaft, or an external coordination service like ZooKeeper in older designs) tracks which broker leads which partition and which brokers are alive, and triggers leader election the moment a leader broker stops heartbeating.
Consumer groups, offset store & retention
Within a consumer group, each partition is consumed by exactly one member, so offset bookkeeping is a simple per-partition counter. Committed offsets live in a durable offset store (itself often a compacted internal topic). A background retention/compaction process reclaims disk by deleting segments older than the retention window or by compacting away superseded values per key.
4. Detailed architecture diagram
The architecture diagram takes every HLD box and answers "how is this actually deployed?" - replica placement across availability zones, the controller quorum, tiered storage, and how a consumer group's instance count maps onto partition count, which is what an interviewer is checking for once they've accepted the high-level shape.
| Decision | Choice | Reasoning |
|---|---|---|
| Partition count vs consumer parallelism | Partitions provisioned above current consumer count (e.g. 6 for order-events) | Partition count is a hard ceiling on parallelism within a group - more consumers than partitions leaves the extras permanently idle, so partition count must be sized for future consumer growth up front. |
| Replication mode | Sync replication with an in-sync-replica (ISR) quorum (acks=all) | The leader only reports a message committed once every ISR member confirms it, which is higher latency but guarantees zero loss even if the leader dies immediately after; async replication (acks=1) is faster but risks losing the last few messages on a leader crash. |
| Retention policy | Time-based deletion by default, log compaction on select topics | Event-stream topics need every event (time-based retention); changelog/state topics only need the latest value per key forever (log compaction) - mixing them wrong either wastes disk or silently drops history that was needed. |
5. Sequence diagrams for the two critical flows
A sequence diagram is where an interviewer checks whether you actually understand call order, which acknowledgment level is being waited on, and what happens on failure between the two.
5.1 Producer publish with acks=all (write path)
Steps 3 and 4 fire in parallel and the leader waits for the slowest ISR member before step 7 - that wait is the entire cost of durability. With acks=1 the leader would return the ack right after step 2, skipping 3-7 entirely and trading durability for lower latency; with acks=0 it would not even wait for step 2's local append to be flushed.
5.2 Consumer poll and offset commit (read path)
Committing the offset only after the batch is fully processed (step 4, after step 3) is what makes delivery at-least-once rather than at-most-once: if the consumer crashes between steps 3 and 4, the committed offset never advances past 118400, so the next poll - by this consumer restarting or by a rebalanced peer - re-fetches and reprocesses the same batch. That is a deliberate trade-off, not a bug: it guarantees no message is silently skipped, at the cost of possible duplicate processing that the consumer's own logic must tolerate.
6. Entity-relationship (ER) diagram and metadata model
This ER diagram models cluster metadata, not the messages themselves - the actual message log lives in purpose-built append-only segment files on disk, never in a database. Metadata has to answer: which partitions belong to which topic, who leads each partition, and where each consumer group currently is per partition.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Message log (topic/partition data) | Purpose-built append-only segment files (Kafka's own log format), never a general-purpose database. | Optimized purely for sequential append and offset-indexed sequential read; a relational engine's transactions and random-I/O indexes would only add overhead the log doesn't need. |
| Cluster & topic metadata | Small enough to live in a Raft-based coordination quorum (KRaft) or even a relational store. | Needs strong consistency and fast propagation - a stale leader pointer sends producers/consumers to the wrong broker until the metadata catches up. |
7. Deep dives interviewers actually probe
Why is a partition the unit of both parallelism and ordering?
A partition is a single ordered, append-only log with exactly one leader broker, so it is both the smallest unit that can be written/read in parallel (more partitions = more concurrent leaders = more throughput) and the largest unit within which ordering is guaranteed (Kafka never guarantees order across partitions). A producer's partitioning key choice sits directly on this trade-off: sending every message for a given entity (e.g. a specific user's events) to the same key guarantees they land in the same partition and are read in order, but if that key is unusually hot it also concentrates load onto one partition's leader, capping that entity's throughput at a single partition's ceiling.
// Typical client-side partitioner
int partitionFor(String key, int numPartitions) {
int hash = murmur2(key.getBytes(UTF_8));
return Math.abs(hash) % numPartitions; // same key -> same partition, always
}What does acks=0 / acks=1 / acks=all actually trade off?
These are the durability/latency knob on the write path. acks=0: producer never waits for any response, sub-millisecond latency, but a message is silently lost if it never even reaches the leader's socket buffer. acks=1: producer waits only for the leader's local append (roughly 1-2ms), but if the leader crashes microseconds after acking - before any follower replicated it - the message is gone even though the producer believes it succeeded. acks=all: producer waits for every in-sync replica (ISR) to confirm (roughly 5-10ms under normal replication lag), and the message survives as long as at least one ISR member survives the failure - this is the only setting that satisfies the "no committed message is ever lost" requirement.
What happens during consumer group rebalancing?
When a consumer joins or leaves a group (deploy, crash, scale-out), the group coordinator must reassign partitions among the surviving members. The classic "stop-the-world" rebalance pauses every consumer in the group - even ones whose partition assignment doesn't change - for the duration of reassignment, which can be seconds on a large group and directly stalls consumption. Incremental cooperative rebalancing instead only revokes the specific partitions that actually need to move to a different consumer, letting unaffected members keep consuming throughout - it trades a more complex two-phase protocol for a much shorter pause window, which matters a lot at high partition/consumer counts.
Why is at-least-once the default, and what would exactly-once cost?
At-least-once falls out naturally from committing the offset after processing (see the crash scenario in the consume sequence diagram above): a crash between processing and committing causes safe reprocessing rather than silent loss, which is the cheaper failure mode to default to. True exactly-once requires two additional mechanisms working together: an idempotent producer (each message carries a producer ID + monotonic sequence number so the broker can drop exact duplicates from retries) and a transactional write that atomically spans both the log append and the offset commit, so a consumer-then-produce chain either fully commits or fully rolls back as one unit. That coordination overhead (extra round trips, transaction markers written to the log) measurably lowers throughput, which is why most high-volume pipelines accept at-least-once plus an idempotent consumer instead of paying for exactly-once everywhere.
What is the single biggest bottleneck as this scales 10x?
Not a single broker's disk in isolation - that's solved by adding partitions and brokers per the capacity math above. At 10x scale (tens of thousands of partitions cluster-wide) the real bottleneck shifts to the controller: every leadership change, broker join/leave, or partition creation must propagate to every broker's metadata cache, and with an external coordination service this metadata-propagation latency grows with partition count, causing slow leader elections and stale-metadata errors during incidents. The two levers that address this are tiered storage (offloading cold segments to object storage so local disk pressure stops scaling with retention window, addressing the I/O side) and moving the controller onto a built-in Raft-based consensus layer (KRaft-style) instead of an external coordination service, which shrinks the metadata footprint and cuts propagation latency at high partition counts.
Post a Comment
Add