System design deep dive · HLD
Design a log aggregation system (ELK-style): full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for log shipping/indexing and full-text search, and an entity-relationship diagram for the metadata model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
A log aggregation system is most stressed exactly when it's most needed: during an incident, when log volume spikes 5-10x and engineers are simultaneously running expensive ad hoc searches - so ingestion must never be the thing that falls over.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide the size of the Kafka buffer needed to survive an incident-driven spike, how many index shards are needed, and how the hot/warm/cold tiering saves cost.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Fleet size | 50,000 hosts/containers, ~10 log lines/sec/host average | ~500,000 log lines/sec average |
| Incident spike | Error-heavy code paths log 5x more during an outage | ~2.5 million log lines/sec peak - this is the number the pipeline must absorb without dropping data |
| Raw ingest volume | ~500 bytes/line average (message + structured fields) | 500K/s × 500B ≈ 250 MB/sec ≈ ~21.6 TB/day raw |
| Indexed size | Inverted index + stored fields typically ~1.1x raw after compression trade-offs | ~24 TB/day added to the hot tier |
| Hot tier retention | 7 days of fast, frequently-queried storage (SSD-backed) | ~170 TB hot tier, sized for sub-second recent-log queries |
| Warm/cold tiers | 30 days warm (spinning disk/cheaper nodes), 1 year cold (object storage, rarely queried) | ~720 TB warm, several PB/year cold - driving the case for tiered storage instead of one uniform cluster |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flow from a log line on disk to a searchable, retained document, without yet committing to shard counts, tiering policy, or cluster topology.
What each box owns
Host agent and collector tier
A lightweight agent (Filebeat/Fluent Bit-style) tails log files or reads container stdout, tracks its own read offset so it can resume after a restart without re-reading or losing lines, and ships batches to a collector over a lossy-tolerant protocol with local disk buffering if the collector is unreachable - this local buffer is what protects the application from ever blocking on logging.
Kafka buffer
Decouples "receiving a log line" from "indexing a log line." During a 5x incident spike, the collector keeps accepting and producing to Kafka at full rate even if the indexing pipeline temporarily falls behind - the buffer's retention window (a few hours) is the slack that lets indexing catch up after the spike passes instead of dropping data.
Parser/enricher pipeline
Applies grok-style pattern matching or structured (JSON) parsing to extract fields like level, service, trace_id, and host, tags the document with the source's known metadata (region, environment), and normalizes timestamps to UTC. Lines that fail to parse are still indexed with the raw message intact and a parse_failed flag, rather than being dropped.
Tiered index storage and query API
New documents land in hot-tier shards on fast storage for the first several days when they're most likely to be queried during active incident response. An index lifecycle policy rolls indices to warm and then cold tiers (cheaper storage, fewer replicas) as they age, and eventually deletes them per retention policy. The query API fans a search out only to the indices whose time range overlaps the query, which is what keeps a "last 15 minutes" search fast even with petabytes retained overall.
4. Detailed architecture diagram
The architecture diagram answers how backpressure is actually handled at the agent, how the search cluster is sharded and tiered, and how rollover/retention runs without downtime - the details an interviewer checks once the HLD shape is accepted.
| Decision | Choice | Reasoning |
|---|---|---|
| Local disk spool on the agent | Buffer-then-drop-oldest, not block-the-application | A logging pipeline outage must never propagate backpressure into production request handling; losing the oldest buffered logs is an acceptable trade-off, silently blocking writes is not. |
| Kafka retention window | ~6 hours, sized to the largest expected indexing outage | Gives the indexing pipeline enough slack to fall behind during a spike and catch up afterward without permanently losing data. |
| Index sharding | Time-based daily indices, not one giant index | Lets old data be dropped by simply deleting whole indices (cheap) instead of running expensive delete-by-query across a monolithic index. |
| Hot/warm/cold tiering | SSD (7d) → HDD (30d) → object storage (1yr) | Most queries target the last few days; paying for SSD-speed storage on data nobody queries anymore would multiply cost for no benefit. |
5. Sequence diagrams for the two critical flows
A sequence diagram is where an interviewer checks whether shipping is decoupled from indexing, and whether a search actually narrows its scope by time before touching data.
5.1 Log shipping and indexing
Step 4 - advancing the agent's local checkpoint only after Kafka acks the batch in step 3 - is what makes the pipeline crash-safe: if the collector or the agent restarts before the ack, the same lines are re-read and re-shipped rather than silently lost, and downstream idempotent handling (or accepted rare duplicates) is preferred over any risk of a gap. Step 8's "refresh interval" is a deliberate small delay (typically ~1s) before a newly indexed document becomes visible to search - a knob traded against indexing throughput.
5.2 Full-text search query across a time range
Step 2 is the single biggest performance lever in the whole query path: because indices are partitioned by day, the query planner can discard every index outside the requested time range before issuing a single shard query, rather than relying on a query-time filter to skip irrelevant data after reading it. Step 4 makes that skip explicit - the warm/cold tiers are never touched for a "last 15 minutes" search, which is exactly why petabytes of retained cold data don't slow down the incident-response query engineers actually run.
6. Entity-relationship (ER) diagram and schema
Unlike the previous systems, the bulk of the data (log_entries) lives in a search engine's inverted index, not a relational table - but the metadata that controls sourcing, indexing, and retention is genuinely relational and worth modeling explicitly.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Inverted-index search engine (Elasticsearch/OpenSearch) for log_entries | Free-text search and relevance ranking across huge volumes is the primary access pattern. | Expensive per-document overhead compared to columnar stores; mapping explosion from unbounded dynamic fields must be controlled. |
| Relational (Postgres) for log_sources and indices metadata | Small tables (thousands to low millions of rows) needing simple queries for lifecycle management and alerting. | Not meant to scale to log volume itself - keep raw log content out of this tier entirely. |
7. Deep dives interviewers actually probe
How do you handle a 5-10x log volume spike during an incident without dropping data?
Two layers absorb it: the agent's local disk spool tolerates a slow or unreachable collector for minutes without blocking the application, and Kafka's multi-hour retention window absorbs the gap between spike-time ingest rate and the indexing pipeline's steady-state throughput. Indexing consumers auto-scale on consumer lag (not on raw ingest rate, which is noisy) so the pipeline catches up within minutes after the spike passes, rather than needing to be provisioned for peak load year-round.
Why time-based daily indices instead of one continuously-growing index?
Deleting expired data becomes dropping a whole index (near-instant metadata operation) instead of a delete-by-query that has to find and mark documents individually across a monolithic index - the latter is slow and fragments the index, requiring expensive background merges. Daily indices also let the query planner skip entire indices outside a query's time range before touching any shard, which is the main reason time-bounded searches stay fast at any retained volume.
What happens to a log line that fails to parse?
It's still indexed - with the raw message stored verbatim and a parse_failed:true field - rather than dropped, because losing visibility into exactly the malformed, often error-adjacent lines during an incident is worse than having an imperfectly structured document. A separate low-volume metric tracks the parse failure rate per source so a service that starts emitting a new, unrecognized log format gets noticed and its parser pattern updated.
How does hot/warm/cold tiering actually save cost without breaking search?
Warm-tier nodes use cheaper spinning disks and drop replica count from 2 to 1 (some durability risk accepted for cost, since this is operational log data, not the system of record). Indices are force-merged into fewer, larger segments once they roll to warm - segment merging is expensive so it only happens once, after writes to that index have stopped. Cold tier moves to object storage entirely, sacrificing query latency for near-zero storage cost, which is acceptable since cold-tier queries are rare, deliberate lookups (compliance/forensics), not the incident-response fast path.
How do you prevent one noisy service from overwhelming the shared cluster?
Per-source ingest quotas at the collector tier (a service producing far above its historical baseline gets rate-limited, with a metric alerting the owning team rather than silently truncating), plus separate Kafka topics per major service so one service's backlog doesn't head-of-line-block another's. On the query side, a per-query timeout and a maximum-indices-touched cap prevent one engineer's overly broad "search everything, all time" query from starving cluster resources needed for active incident response.
Post a Comment
Add