System design deep dive · HLD
Design a Distributed File System (HDFS-style): full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for block-write replication and failure-triggered re-replication, and an entity-relationship diagram for the metadata model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
A distributed file system trades general-purpose file semantics for extreme scale and durability - the first thing to pin down out loud is that this is write-once/append, large-sequential-file oriented, not a POSIX-compliant random-access filesystem.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide the two hardest constraints in the system: whether the entire block map fits in the NameNode's RAM, and how much cross-rack bandwidth the replication pipeline consumes.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Cluster raw capacity | 10,000 DataNodes × ~10 TB usable each | ~100 PB raw cluster capacity |
| Block count | 128MB blocks, 100PB stored, replication factor 3 | ~800M unique blocks × 3 = ~2.4B block replicas cluster-wide |
| NameNode metadata size | ~150 bytes/block for block map + ~200 bytes/file for inode | ~800M blocks × 150B ≈ 120GB - must fit entirely in NameNode RAM |
| Write throughput per client | 3-way replication pipeline, 1 Gbps NIC per DataNode | ~100-120 MB/s sustained per client stream |
| Heartbeat / failure detection | Heartbeat every 3s, DataNode marked stale after 3 missed, dead after 10 missed | Detection window of ~30s before re-replication of that node's blocks begins |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flow for both the write path (client to DataNodes, with the NameNode only brokering locations) and the read path, without yet committing to HA failover mechanics or rack topology.
What each box owns
NameNode (metadata plane)
Holds the entire namespace tree (directories, filenames, permissions) and the block map (which blocks make up each file and which DataNodes hold each block) - entirely in RAM for speed. Critically, file data never flows through the NameNode; it only ever hands out block locations, so it never becomes a data-path bottleneck.
Standby NameNode + JournalNodes
Every namespace mutation is written to a quorum of JournalNodes before being applied, and a hot standby NameNode continuously tails that journal so it can take over within seconds of an active-NameNode failure - eliminating the metadata single point of failure that early distributed file systems were criticized for.
DataNodes (data plane)
Store blocks as ordinary files on local disks, serve read/write requests directly to clients (bypassing the NameNode entirely for the actual bytes), and send a heartbeat plus periodic full block report to the NameNode so it can maintain an accurate view of where every replica lives.
Heartbeat channel and re-replication queue
When a DataNode's heartbeats stop, the NameNode marks it dead and immediately enqueues every block that node held as under-replicated; a background re-replication process then copies each under-replicated block from a surviving replica to a healthy DataNode, restoring the replication factor without any client-visible interruption.
4. Detailed architecture diagram
The architecture diagram adds the piece that most differentiates a real deployment from the HLD: rack topology, and the specific rule that governs where each of the 3 replicas of a block is placed.
| Decision | Choice | Reasoning |
|---|---|---|
| Replica placement | 2 replicas same rack (different nodes), 1 replica different rack | Balances write bandwidth (2 nearby replicas fill fast over the top-of-rack switch) against durability (surviving a whole-rack power/network failure). |
| Metadata architecture | Single active NameNode holding all metadata in RAM, HA standby via JournalNode quorum | Keeps every metadata operation a fast in-memory lookup; HA solves the availability risk without sharding metadata, which real deployments avoid unless namespace size forces federation. |
| Block size | 128MB (configurable up to 256MB) | Large blocks amortize seek overhead for sequential scans and keep the NameNode's metadata table small enough to fit in RAM at petabyte scale. |
| Cold-data redundancy | Erasure coding (Reed-Solomon 6+3) for rarely-accessed files instead of 3x replication | Cuts storage overhead from 200% to ~50% for data that is read rarely enough that erasure coding's higher reconstruction cost on failure is an acceptable trade. |
5. Sequence diagrams for the two critical flows
A sequence diagram here is where an interviewer checks whether replication is understood as a client-driven pipeline (not something the NameNode does), and whether failure detection is separated cleanly from the re-replication it triggers.
5.1 File write with 3-way replication pipeline
Bytes flow client → DN1 → DN2 → DN3 in a single forward pipeline rather than the client uploading three separate copies - this uses the client's upload bandwidth once instead of three times. The NameNode is only consulted once at the start (step 1-2) to choose which three DataNodes should hold this block; it is never in the data path itself, which is exactly why it can serve metadata for the whole cluster from a single process.
5.2 DataNode failure detection and re-replication
Detection (steps 1-2) and remediation (steps 3-7) are deliberately separate stages: the NameNode's only job at failure time is to notice and mark the node dead cheaply, while the actual re-replication work - the expensive part - is handed to a background queue that copies blocks between DataNodes directly, never through the NameNode. This queue is rate-limited so that a single rack failure re-replicating thousands of blocks at once cannot saturate the cluster's cross-rack network - the same throttle referenced in the architecture diagram above.
6. Entity-relationship (ER) diagram and schema
Unlike a typical relational data model, this "schema" lives almost entirely in the NameNode's in-memory structures rather than a disk-backed database - but it still has to answer the same three questions: what uniquely identifies a block, how block-to-file ordering is preserved, and how block-to-DataNode location is tracked and kept current.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| In-memory NameNode structures, journaled to disk/quorum | Metadata operations must be sub-millisecond and the whole namespace must fit in one process's RAM. | Namespace size is capped by a single NameNode's RAM - very large clusters need HDFS Federation (multiple independent NameNodes, each owning a namespace slice) rather than a bigger box. |
| Relational DB for the namespace (an alternative some object stores take) | You want ad-hoc queries over file metadata (find all files by owner, by size) without exporting fsimage. | Adds a network hop to every metadata operation, which is exactly the latency HDFS's design avoids by keeping metadata in-process. |
7. Deep dives interviewers actually probe
Why use a large block size (128-256MB) instead of the 4KB blocks a local filesystem uses?
Two independent reasons stack together: first, large blocks amortize the seek/disk-head-movement overhead of sequential scans, which is the dominant access pattern for the analytics jobs this system targets. Second, and more decisive at scale, a smaller block size would multiply the number of metadata entries the NameNode must hold in RAM by tens of thousands - at 4KB blocks, the 800M blocks in the capacity table above become on the order of 25 trillion, which no single process's memory could ever hold.
Why is rack-aware placement (2 replicas same rack, 1 different rack) the default, not 3 different racks?
Three different racks would maximize durability but pay for it on every single write: two of the three replica copies would have to cross the (slower, often oversubscribed) inter-rack network. Two-same-rack-one-different is the balance point - the write pipeline's first two hops stay on the fast top-of-rack switch, while the third hop still guarantees the file survives losing an entire rack (power, top-of-rack switch, or a whole row of hardware failing together).
How does the NameNode's metadata scale past what fits in one machine's RAM?
HDFS Federation: instead of sharding the block map (which would require every read to know which shard owns a given block ahead of time, defeating the point), the namespace itself is partitioned into independent volumes, each with its own NameNode, while all NameNodes share the same pool of DataNodes underneath. A client's path prefix determines which NameNode to talk to - this is closer to horizontally partitioning by directory tree than by hash, which keeps directory-level operations (rename, permissions) cheap within a single NameNode.
What consistency model does this actually provide - can a reader see a partial write?
The classic model is write-once-read-many with no in-place random writes: a file being written is not visible to readers as a complete file until it's explicitly closed, though newer versions do expose a stream of already-flushed bytes to concurrent readers. There is no update-in-place semantic at all - "modifying" a file means writing a new file (or, in append-only mode, appending new blocks) - which is a deliberate simplification that removes the entire class of concurrent-writer conflict resolution a POSIX filesystem has to solve.
What's the single biggest bottleneck as the cluster scales 10x?
Not raw storage or DataNode count - those scale close to linearly by adding more nodes. The real bottleneck is the single active NameNode's metadata throughput and RAM ceiling: every namespace operation (open, create, rename, list) still funnels through one process's lock, and its RAM caps total addressable blocks. That's solved either by Federation (splitting the namespace across multiple NameNodes) or by moving to a fully disaggregated metadata service that shards the block map itself - a much larger architectural change that trades the simplicity of "one process holds the whole namespace" for horizontal metadata scalability.
Post a Comment
Add