System design deep dive · HLD
Design an ad click aggregation system: full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for click ingestion and streaming aggregation, and an entity-relationship diagram for the data model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
Ad click aggregation lives at the intersection of two hard problems: absorbing enormous, spiky write throughput, and producing counts advertisers will trust enough to pay an invoice against - which means correctness under duplication and fraud matters as much as speed.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide whether Kafka partitioning can keep up, how much state a streaming aggregation job must hold in memory per window, and how large the raw event data lake grows.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Daily click volume | 10 billion ad clicks/day across the network | ~115,000 clicks/sec average, ~345,000 clicks/sec peak (3x) |
| Raw click event size | ~250 bytes (campaign_id, ad_id, click_id, ip/device hash, ts, referrer) | 10B/day × 250B ≈ 2.5 TB/day raw before compression |
| Active campaigns | ~2 million concurrently active campaigns | ~2M distinct counters updated per 1-minute window at peak load |
| Fraud filter rate | Industry-typical invalid traffic rate ~15-20% | ~1.5-2B clicks/day flagged and excluded from billable aggregates, still retained raw for audit |
| Streaming state size | Per-campaign per-minute counters, ~2M active campaigns × ~100 bytes | ~200 MB of hot aggregation state - fits comfortably in a stream processor's in-memory/RocksDB state store |
| Raw log retention | 2.5 TB/day × 400 days (billing dispute window) | ~1 PB in cold object storage, compressed/columnar |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flow from a raw click to a trustworthy per-minute count, without yet committing to partition counts, region layout, or the specific stream processing engine.
What each box owns
Ingestion service
The single entry point for click events fired by ad servers' redirect endpoints. It validates the payload, assigns/verifies a click_id used for downstream dedup, and produces the event onto Kafka keyed by campaign_id - it deliberately does no aggregation itself, so it can scale purely by adding stateless replicas.
Fraud scoring
Consumes the same click stream and tags each event valid/invalid using a mix of fast rules (click velocity per IP/device within a short window, known datacenter IP ranges, missing device fingerprint) and an offline-trained ML model score. It annotates events rather than dropping them outright, so flagged clicks remain in the raw log for audit and appeal even though they're excluded from billable aggregates.
Stream aggregator
A stateful stream processing job (Flink/Kafka Streams style) that groups valid events into 1-minute tumbling windows keyed by campaign_id, maintains a running count in local state, and emits/updates the aggregated counter store when a window closes - handling out-of-order and slightly late events via watermarking rather than assuming perfectly ordered arrival.
Aggregated counters and the raw data lake
The counters store (Redis for the current/hot minute, an OLAP columnar store like Druid/ClickHouse for historical rollups) is what advertiser dashboards read - it is never treated as billing-authoritative. The raw click data lake retains every event, valid or not, and a separate nightly batch job recomputes the definitive billed count from it, reconciling any drift from the real-time path.
4. Detailed architecture diagram
The architecture diagram answers how Kafka partitioning survives a viral campaign, how stream processor state is checkpointed, and how the real-time and batch paths stay reconcilable - the details an interviewer checks once the HLD shape is accepted.
| Decision | Choice | Reasoning |
|---|---|---|
| Kafka partitioning key | campaign_id, with salted sub-keys for outlier campaigns | Keeps all events for one campaign ordered together for windowed aggregation, while a salt prevents a single viral campaign from overwhelming one partition. |
| Lambda-style dual path | Real-time stream aggregation for dashboards + nightly batch recompute for billing | Real-time counts are useful but approximate under late data and evolving fraud rules; a deterministic batch recompute over immutable raw logs is what actually gets invoiced. |
| Exactly-once semantics | Flink checkpointing + idempotent counter updates keyed by click_id | Kafka only guarantees at-least-once delivery; the stream processor's checkpoint/replay model combined with idempotent state updates is what prevents double-counting on failure recovery. |
| Fraud filtering is tag, not drop | Invalid clicks stay in the raw log, excluded only from billable aggregates | Advertisers dispute fraud calls; the audit trail must exist to investigate and, if needed, reverse a false positive. |
5. Sequence diagrams for the two critical flows
A sequence diagram is where an interviewer checks whether a candidate understands that ingestion, fraud scoring, and aggregation are decoupled asynchronous stages rather than one synchronous call chain.
5.1 Click ingestion and durable logging
Step 6 - redirecting the user to the advertiser's page - happens the instant Kafka acknowledges the write, not after fraud scoring or aggregation, because the user must never notice the ad network's internal pipeline. Step 7 is drawn as async precisely because archival to the data lake can lag by seconds without affecting anything on the user-facing critical path.
5.2 Streaming fraud filter and windowed aggregation
Step 5 is the crux of streaming aggregation: a watermark (the aggregator's estimate of "events older than this have all arrived") allows the window to stay open briefly for network-delayed events before closing, rather than either closing too early (undercounting) or waiting indefinitely (never emitting). Step 8's explicit "as of" timestamp is what tells the dashboard the count is a live approximation, not the final billed number.
6. Entity-relationship (ER) diagram and schema
The data model has to answer: how is a duplicate click detected without scanning history, how is a per-minute rollup queried cheaply for a dashboard, and how does the raw log stay separable from the aggregate for billing disputes.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Columnar data lake (S3 + Parquet) for click_events | Volume is enormous (billions/day) and access is mostly big sequential batch scans for billing, not point lookups. | Not suited for low-latency point queries - the real-time counters store exists specifically to avoid querying this tier for dashboards. |
| OLAP store (Druid/ClickHouse) for aggregated_counters | Dashboards need fast group-by/time-range queries across millions of campaigns. | Eventually consistent with the stream; must be clearly labeled as non-authoritative for billing to avoid disputes. |
7. Deep dives interviewers actually probe
How do you get exactly-once counting when Kafka only guarantees at-least-once?
Two layers of defense: the ingestion service's dedup cache rejects an obviously repeated click_id before it ever reaches Kafka, and the stream processor's checkpointing (Flink-style) makes state updates transactional with the consumer offset - if the job crashes and replays from the last checkpoint, the counter update for an already-processed event is skipped because the offset commit and the state mutation are atomic. The combination converts "delivered at-least-once" into "counted exactly-once."
How does the system handle a click event that arrives 3 minutes late?
The aggregator tracks a watermark - its running estimate of the latest timestamp it can consider "complete" - and keeps a window open for a bounded grace period (e.g. 2 minutes) past its nominal end before emitting. An event later than the grace period is routed to a "late events" side output rather than silently dropped or incorrectly reopening a closed window, and the nightly batch job (which has no such time pressure) is what ultimately incorporates every late event into the authoritative billed count.
How is bot/click-farm traffic actually detected?
Layered signals: velocity rules (more than N clicks from one device/IP within a short window), known datacenter/proxy IP range lists, missing or inconsistent device fingerprints, and an offline-trained model scoring subtler patterns (unnatural click timing distributions, conversion-rate anomalies per source). Rules catch cheap, obvious fraud in real time; the ML score, refreshed periodically from labeled outcomes, catches what static rules miss - both write to the same fraud_flag rather than being two separate pipelines.
What happens when one campaign goes viral and floods a single Kafka partition?
Partitioning strictly by campaign_id caps a single campaign's throughput at whatever one partition (and its consumer) can handle. The fix used here is key salting for outlier campaigns above a throughput threshold: the campaign's events are spread across a handful of sub-keys (campaign_id + hash(click_id) % N), consumed in parallel, and the aggregation operator merges the sub-key counts back into one campaign-level total at window-close - trading a small amount of extra merge logic for horizontal scalability on hot keys.
Why keep a separate nightly batch recompute instead of trusting the stream?
The stream aggregator optimizes for low latency under an approximate watermark and a fraud model that may be revised after the fact (e.g. a fraud ring identified a day later). The batch job re-runs over immutable raw logs with the final, possibly-updated fraud rules and no time pressure, producing a deterministic, reproducible number - this is the number advertisers are actually billed against, and any real-time/batch discrepancy is itself a monitored signal for pipeline bugs.
Post a Comment
Add