System design deep dive · HLD
Design a multi-channel notification system (push / email / SMS): full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for the send path and the retry/backoff path, and an entity-relationship diagram for the data model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
A notification system's defining tension is that it must both never spam a user who opted out and never silently drop a transactional message like a password reset - those two failure modes have opposite costs.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide whether a single queue is enough, why transactional and bulk traffic must be physically separated, and how large the dedup cache needs to be.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Total notifications | 5 billion/month across all channels (transactional + marketing) | ~1,930 sends/sec average |
| Peak campaign burst | A single marketing blast enqueues 50M sends in 20 minutes | ~42,000 enqueues/sec peak, absorbed by queue depth not synchronous processing |
| Transactional traffic | OTP/receipt/alert sends, ~5% of total, latency-sensitive | ~250M/month, ~96 sends/sec average, needs its own priority lane |
| Delivery log storage | ~300 bytes/attempt (channel, provider, status, timestamps), incl. retries | 5B/mo × 1.2 (retry factor) × 300B ≈ 1.8 TB/month |
| Dedup cache | Idempotency key TTL of 24h, ~170M unique triggers/day | ~170M keys × ~80 bytes ≈ 14 GB - fits a Redis cluster |
| Provider rate limits | SMS provider caps ~2,000 msgs/sec per account | requires client-side rate limiting/sharding across provider accounts to avoid 429s during bursts |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flow from a trigger to a delivered (or failed) notification, without yet committing to region counts, provider accounts, or queue partitioning.
What each box owns
Notification API
The single ingest point for every internal service. It computes an idempotency key (typically a hash of trigger type + user_id + template + a caller-supplied dedup token), checks the dedup cache, and rejects duplicates before any expensive work happens. It never talks to a channel provider directly.
Preference service
Owns per-user opt-outs by category (marketing, product-updates, security-alerts), quiet-hours windows, and channel priority order. Security/transactional categories are marked non-suppressible so a password-reset email can never be silently dropped by a marketing opt-out, which is the specific policy that keeps the two failure modes from the requirements section from colliding.
Template render
Loads the versioned template for the given notification type and renders channel-specific output (push payload, HTML email, SMS text) by substituting variables from the trigger payload. Keeping rendering separate from the API means template edits deploy independently of the ingest path.
Channel queues, workers, and delivery log
Each channel has its own queue so a Twilio outage cannot back up email or push. Workers call the provider, respecting its rate limit, and write a delivery_logs row with the final status. Failures route to a retry/backoff queue rather than being dropped, and repeated failures eventually land in a dead-letter queue for alerting.
4. Detailed architecture diagram
The architecture diagram answers how priority separation, provider rate limits, and retry/backoff are actually deployed - the details an interviewer checks once the HLD shape is accepted.
| Decision | Choice | Reasoning |
|---|---|---|
| Priority separation | Physically separate queues for transactional vs bulk | A 50M-message marketing blast must never delay a password-reset email; separate queues mean the two workloads can't contend for the same worker pool. |
| Provider rate limiting | Token-bucket limiter sharded across multiple provider accounts | Providers hard-cap throughput per account (e.g. ~2,000 SMS/sec); sharding across accounts is the only way to sustain campaign-level burst without 429s. |
| Retry strategy | Exponential backoff, capped at 5 attempts, then DLQ | Transient provider errors (5xx, timeouts) resolve within minutes; permanent errors (invalid number, hard bounce) should fail fast into the DLQ rather than retry forever. |
| Provider abstraction layer | A thin interface per channel, not a direct SDK call from workers | Lets a channel's provider be swapped (e.g. Twilio to a backup SMS vendor) without touching queueing, dedup, or preference logic. |
5. Sequence diagrams for the two critical flows
A sequence diagram is where an interviewer checks whether a candidate actually enforces preferences before rendering content, and whether failure handling is a first-class path rather than an afterthought.
5.1 Trigger a notification and fan out across channels
Step 2 - the dedup check - happens before any preference lookup or rendering work, specifically so a retried trigger costs almost nothing. Note that SMS is silently excluded in step 4 because the user opted out of that channel for this category; the caller in step 8 gets a 202 immediately and never learns (or needs to know) which channels were actually used - that detail lives in the delivery log.
5.2 Delivery failure, retry with backoff, and dead-lettering
Every attempt is logged in step 4 regardless of outcome, which is what makes the delivery_logs table a complete audit trail rather than only recording final status. The backoff delays in step 6 double each time specifically to avoid hammering a degraded provider right as it's recovering; after the fifth failure the message is dead-lettered rather than retried indefinitely, converting an invisible silent failure into a paged, actionable alert.
6. Entity-relationship (ER) diagram and schema
The data model has to answer: how is a duplicate trigger detected cheaply, how does a versioned template stay decoupled from a specific send, and how is per-attempt delivery status queried without scanning the whole history.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Relational (Postgres), notifications + templates + preferences | You need the unique constraint on idempotency_key and simple joins for the delivery-status API. | delivery_logs volume (billions of rows/month) will outgrow a single relational table without partitioning/archival. |
| Wide-column (Cassandra/DynamoDB) for delivery_logs specifically | Write volume and retention window (90 days) matter more than ad hoc joins for this one table. | Querying "all attempts for this notification" needs the partition key chosen up front (notification_id), or it requires a scan. |
7. Deep dives interviewers actually probe
How exactly is a duplicate trigger detected?
The idempotency key is a hash of (trigger type, user_id, template_id, and a caller-supplied dedup token - e.g. order_id for a shipping notification). The Notification API checks this key against Redis with a SET NX before doing anything else; if the key already exists, the request is accepted and immediately discarded as a duplicate. The unique constraint on the notifications table is a second, durable line of defense in case two API instances race on the same key within the same millisecond.
Why exponential backoff instead of fixed-interval retry?
A fixed short interval (e.g. retry every 10s) against a provider that's actively degraded just adds more load to a system that's already failing, potentially delaying its recovery. Exponential backoff (30s, 2m, 8m, 32m) spaces retries out specifically so that by the time later attempts happen, a transient provider blip has likely resolved - and it bounds the total retry window so a permanently-broken address doesn't retry for days.
How is a marketing opt-out enforced without ever blocking a security alert?
Every template is tagged with a category, and category is what user_preferences keys on - not the notification as a whole. Security and transactional categories are configured as non-suppressible at the template level, so the preference check for those categories always returns "allowed" regardless of what the user has opted out of, which is a policy decision made explicit in code rather than left to a runtime edge case.
How do you avoid a 50M-message campaign starving normal traffic?
Separate queues per priority lane (section 4) mean the bulk lane's depth is invisible to the transactional lane's workers. Within the bulk lane itself, the queue is sharded into partitions so a single campaign's messages spread across many workers rather than serializing behind one queue's throughput ceiling, and workers auto-scale based on queue depth rather than running a fixed pool sized for average load.
What happens if the chosen SMS provider has an outage?
Because workers call providers through the abstraction layer rather than a hardcoded SDK, a circuit breaker on the primary provider can fail over to a secondary provider account (or vendor) for new sends once error rates cross a threshold, while in-flight retries for the primary continue draining independently - this is why provider identity is a field on delivery_logs rather than an assumption baked into the channel.
Post a Comment
Add