System design deep dive · HLD
Design a Two-Factor Authentication (2FA) system: full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for TOTP verification and SMS/push OTP delivery, and an entity-relationship diagram for the data model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
A strong system design answer starts by pinning down scope out loud. For 2FA, that means separating "prove possession of a second factor" from "what happens when the user loses that factor" - recovery is at least as important as verification and is where most real-world incidents happen.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide whether the verify path can run entirely off a cache/replica read, how big the SMS delivery bill is, and how much the rate limiter needs to hold in memory per user.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Enrolled users | 200 million with 2FA enabled | ~200M rows in otp_secrets |
| Logins requiring 2FA/day | ~15% of 500M daily logins | ~75M verify calls/day ≈ 870/sec average, ~8,000/sec peak (morning login surge) |
| SMS/push OTP sends | ~20% of verifies use SMS/push instead of TOTP app | ~15M SMS/push sends/day, at ~$0.01-0.05/SMS a meaningful direct cost driver |
| Storage per secret record | ~300 bytes (encrypted seed, algorithm, counter/step metadata) | 200M × 300B ≈ 60 GB - comfortably fits a replicated relational cluster |
| Rate-limit state | Sliding window counter per user, ~100 bytes each | ~200M × 100B ≈ 20 GB hot working set - fits a Redis cluster |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flow between them, without committing yet to specific infrastructure, regions, or replica counts - that level of detail belongs in the architecture diagram in the next section.
What each box owns
Enrollment service (write path)
Generates a random 160-bit TOTP seed, wraps it via envelope encryption (a data key from the KMS encrypts the seed, the KMS key itself never leaves the KMS), renders the provisioning QR code (otpauth:// URI), and issues 10 single-use backup codes whose salted hashes - never the plaintext - are stored. It never talks to the OTP delivery gateway; SMS enrollment is a separate, lighter-weight phone-number-verification flow.
Verification service (read/verify path)
Decrypts the stored seed via the KMS, computes the expected TOTP for the current time step (and the adjacent ±1 step to tolerate clock drift), and compares it to the submitted code using a constant-time comparison to avoid timing side-channels. Every attempt - success or failure - first passes through the rate limiter, and a failure increments a per-user failure counter used for both rate limiting and account-lock alerting.
KMS and secret store
The secret store never holds a plaintext TOTP seed; it holds ciphertext plus a reference to the KMS data key used to wrap it. This means a database dump alone is useless to an attacker without also compromising the KMS - the two systems are operated with separate access controls specifically so one breach doesn't imply the other.
Rate limiter, OTP delivery gateway, and audit log
The rate limiter enforces both a per-user cap (e.g. 5 verify attempts/minute) and a per-IP cap (to blunt distributed credential-stuffing), backed by Redis for O(1) sliding-window checks. The OTP delivery gateway abstracts away the specific SMS/push provider behind a common interface so a provider outage can fail over to a secondary. The audit log records every enrollment, verify attempt, and backup-code use asynchronously, so a slow audit write can never add latency to the login path.
4. Detailed architecture diagram
The architecture diagram takes every HLD box and answers "how is this actually deployed?" - replica counts, sharding, regions, and the specific technology choice, which is what an interviewer is checking for once they've accepted the high-level shape.
| Decision | Choice | Reasoning |
|---|---|---|
| Secret replication scope | KMS keys and Redis rate-limit state stay regional, not cross-region replicated | Minimizes the blast radius of a single region's key material being compromised; a user's secret is decrypted only where it was enrolled or via a controlled re-key migration. |
| Secret store sharding key | Hash of user_id | Even distribution; the verification service always has user_id at verify time, so no secondary lookup is needed to find the shard. |
| OTP delivery provider strategy | Provider router with automatic failover, not a single hard-coded SMS vendor | A single carrier/provider outage must not lock every SMS-based user out; the router shifts traffic to a secondary provider transparently. |
| Audit isolation | Separate append-only store, written asynchronously | A backlog in fraud/anomaly processing must never add latency to the synchronous verify path that gates login. |
5. Sequence diagrams for the two critical flows
A sequence diagram is where an interviewer checks whether you actually understand call order, what is synchronous versus fire-and-forget, and exactly where rate limiting sits relative to the actual verification logic.
5.1 TOTP verification during login
Step 3's rate-limit check happens before any decryption or comparison work runs, so a client that has already exhausted its attempt budget is rejected cheaply without ever touching the KMS - protecting both the user's account and the KMS's request quota from a brute-force loop. Step 6's ±1 time-step tolerance (90 seconds total) absorbs realistic clock drift on the user's phone without meaningfully widening the guessable window.
5.2 SMS/push OTP delivery with rate limiting
Only the OTP's salted hash is stored (step 4), the same principle as a password, so a database read can never directly reveal a still-valid code. Step 6-7's provider failover is why the OTP is generated and persisted before any provider is contacted - if the send itself fails and needs to be retried through a different provider, the code stays valid and does not need to be regenerated, which would otherwise invalidate a code the user might already be looking at from a slow first delivery.
6. Entity-relationship (ER) diagram and schema
The data model has to answer three questions: how is a TOTP seed protected at rest, how is a brute-force attempt tracked without becoming the write bottleneck, and how does backup-code consumption stay atomic so a code can never be used twice.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Relational (Postgres), sharded by user_id | You want row-level encryption at rest plus transactional "verify and mark backup code used" atomicity. | Cross-shard analytics (e.g. global fraud queries) need a separate read-optimized replica or warehouse. |
| Key-value / wide-column (DynamoDB/Cassandra) | Access pattern is purely "get secret by user_id" and "append attempt event" at very high scale. | Atomic backup-code consumption needs conditional writes; ad-hoc joins for support tooling are harder. |
7. Deep dives interviewers actually probe
TOTP vs HOTP - why time-based, and how is clock drift handled?
TOTP (RFC 6238) derives the code from the shared seed and the current 30-second time step; HOTP (RFC 4226) derives it from the seed and a monotonically increasing counter. TOTP is preferred because it self-resyncs every 30 seconds without any server-side counter state to track per device, while HOTP's counter can desync if a code is generated but never submitted (common on hardware tokens with a physical button). The verify service checks the current step and one step on either side (±30s) to tolerate realistic clock drift on the user's device without materially widening the brute-force window.
// TOTP core (RFC 6238): HMAC-SHA1 over the time counter, truncated to 6 digits
long timeCounter = Instant.now().getEpochSecond() / 30;
byte[] hmac = HmacSHA1(seed, timeCounter);
int code = truncate(hmac) % 1_000_000;Why is rate limiting "load-bearing" for a 6-digit code?
A 6-digit code has only 1,000,000 possible values, and any 30-second window has a valid code. Without rate limiting, an attacker could plausibly brute-force it in well under a minute against an unthrottled endpoint. A cap like 5 attempts/minute per user (plus a separate per-IP cap to blunt distributed attempts) reduces the practical success probability to effectively zero within the code's validity window, which is why the rate limiter sits in front of the cryptographic comparison, not behind it.
Backup-code recovery - how do you prevent reuse and abuse?
Each of the 10 issued backup codes is single-use: consuming one is a conditional UPDATE that flips its stored hash's status from unused to used only if it was still unused, making concurrent double-submission safe. After a backup code is used, the user is prompted to re-enroll a fresh TOTP factor and the remaining backup codes are optionally invalidated if the login pattern looks anomalous (new device, new geography), treating "used a backup code" itself as a moderate-risk signal worth extra scrutiny.
What happens when the SMS provider is down or a number is ported (SIM-swap risk)?
The provider router fails over to a secondary SMS/push vendor automatically (sequence 5.2), bounding provider-outage impact. SIM-swap is a distinct threat: because SMS OTP is inherently phone-number-bound, high-risk actions (changing the registered phone number itself, or disabling 2FA) require re-verifying via the existing factor first and impose a cooldown period before the new number becomes usable for recovery, specifically to blunt "port the number, then immediately use it to take over the account" attacks.
How do you protect the TOTP seed if the database is ever dumped?
Envelope encryption: each seed is encrypted with a per-record or per-tenant data key, and that data key is itself encrypted by a master key that lives only in the KMS and never touches application memory in plaintext form for longer than a single decrypt operation. A database dump yields only ciphertext plus opaque key references - useless without also compromising the KMS, which is operated under separate credentials and audit logging specifically so the two breaches are not correlated.
What is the single biggest bottleneck as this scales 10x?
Not TOTP verification - it is cheap, stateless HMAC computation once the seed is decrypted. The real bottleneck becomes KMS decrypt call volume, since most KMS providers charge and rate-limit per API call. That is solved by caching a short-lived (single-request-scoped, in-memory only, never persisted) decrypted seed within the verify request's execution context, and by batching KMS calls where the provider supports it, rather than increasing KMS request quota indefinitely.
Post a Comment
Add