System design deep dive · HLD
Design a Recommendation System (Amazon-style "customers also bought"): full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for offline training and online serving, and an entity-relationship diagram for the data model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
A strong system design answer starts by pinning down scope out loud. For a recommendation system, that means separating "what gets recommended and how" from "how well it must perform," and stating the scale assumptions (catalog size, event volume) that every later diagram depends on.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide almost everything downstream: whether brute-force scoring is even conceivable, how big the vector index gets, how much Kafka throughput the feedback loop needs, and how large the serving cache must be.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Recommendation requests | 500M page views/day need a rec slot | 500M ÷ 86,400s ≈ 5,800 req/s average, ≈17,000 req/s peak (×3) |
| Item embedding index size | 350M items × 128-dim float32 (4B/dim) | 350M × 128 × 4B ≈ 179 GB raw vectors; ≈270-360 GB once the HNSW graph overhead is added |
| Interaction events (feedback loop) | Clicks + purchases + add-to-cart, ≈5B events/day | 5B ÷ 86,400s ≈ 58,000 events/s average; Kafka sized for ≈150,000 events/s peak |
| Offline training dataset / run | Trailing 60-day window of interaction logs | ≈300B raw events, downsampled/negative-sampled to ≈20-30B training rows (≈5-10 TB compressed Parquet) |
| Online rec cache size | Top-50 cached items/user × 300M users | ≈1 KB/user (ids + scores) × 300M ≈ 300 GB across a sharded Redis cluster |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flow between them - the online serving path that answers a request, and the offline/feedback loop that keeps embeddings and the ranker up to date - without committing yet to replica counts or regions.
What each box owns
Rec service (orchestrator)
Receives the request (a user id and/or an item id), checks the rec cache first, and on a miss fans out to candidate generation and then ranking. It owns the overall latency budget and is what enforces "return a decent list within the deadline, even if a downstream step is slow" via short timeouts and fallback to popularity-based results.
Candidate generation (ANN over item embeddings)
Takes the query item's or user's embedding vector and runs an approximate nearest-neighbor search over the embedding store to narrow 350 million items down to a few hundred plausible candidates in single-digit milliseconds. It deliberately does not try to be precise - recall of "the true top-K is probably in here somewhere" is enough, because the ranking service will re-score this small set precisely.
Ranking service (learned ranker)
Scores the few hundred candidates with a model that can afford to be expensive per-item (a two-tower network or gradient-boosted tree) because it only runs on candidates, not the whole catalog. It pulls user, item, and context features from the online feature store at request time - things like the user's last-viewed categories and the item's live popularity score.
Feature store: online vs offline split
A Feast-style split: an offline store (data lake / warehouse) holds the full historical feature history used to build training datasets, while an online store (Redis/DynamoDB) holds only the latest value of each feature, optimized for single-digit-millisecond point lookups at request time. The same feature definitions are used in both places so training and serving never see a different computation for the "same" feature (train/serve skew).
Rec cache, event ingestion, and the offline training loop
The rec cache holds precomputed ranked lists per user/context so most requests never touch candidate generation or ranking at all. Event ingestion publishes every click, purchase, and add-to-cart to Kafka without blocking the page. The offline training pipeline runs on a schedule, retrains embeddings and the ranker from that event history, and pushes new artifacts to the embedding store and ranking service - this is the loop that makes recommendations improve over time.
4. Detailed architecture diagram
The architecture diagram takes every HLD box and answers "how is this actually deployed, and how does feedback get from a click back into a ranking decision?" - which is what an interviewer is checking for once they've accepted the high-level shape.
| Decision | Choice | Reasoning |
|---|---|---|
| Retrieval strategy | Two-stage: ANN candidate generation, then a learned ranker | Scoring all 350M items with a full ranking model per request is computationally infeasible inside a 100ms budget; ANN narrows to ~500 cheaply, so the expensive model only runs on that small set. |
| ANN index algorithm | HNSW for the hot catalog tier; IVF + product quantization for the long tail | HNSW gives high recall at low latency but keeps full-precision vectors in memory (~300GB); IVF+PQ compresses vectors 8-16×, letting rarely-viewed items fit in memory affordably at a small recall cost. |
| Retraining cadence | Nightly/batch retraining, not continuous online learning | Billions of daily events make synchronous online learning hard to validate and easy to regress silently; nightly batch gives a stable, offline-validated checkpoint, while the online feature store bridges the freshness gap. |
| Rec cache TTL | 15-30 minute TTL, refreshed by a background job | Precomputed caching is what makes <100ms serving affordable at 300M-user scale; a longer TTL trades some staleness for far fewer expensive ranking calls, while recent clicks still nudge in-session re-ranking via the feature store. |
5. Sequence diagrams for the two critical flows
A sequence diagram is where an interviewer checks whether you actually understand call order, what is synchronous versus fire-and-forget, and how the offline and online halves of the system connect.
5.1 Offline training and embedding pipeline (batch)
Steps 4-5 are the guardrail: a candidate model that doesn't beat the current production model on an offline metric never reaches step 6. Steps 8-9 are drawn dashed because the rollout to serving nodes is a controlled canary, not an instant global swap - only after the canary's live guardrail metrics (click-through, revenue-per-session) hold up does traffic ramp from 5% to 100%.
5.2 Online request for recommendations, plus the async feedback event
On a cache hit, steps 2-6 collapse entirely and the response returns from the cache alone in a few milliseconds - the common case for a returning user's homepage. Step 8 is drawn dashed and gray for the same reason the URL shortener's click log is: it is published and never awaited, so a slow or backed-up stream processor can never add latency to the page. A separate stream processor (shown in the architecture diagram) consumes that event and updates the online feature store, which can influence the *next* request's ranking or an in-session re-rank, but never this one.
6. Entity-relationship (ER) diagram and data model
The data model has to answer three questions: where do per-user features live so the ranker can read them in a millisecond, how are item embeddings stored so an ANN index can serve them, and how is a cached recommendation list keyed so the right list comes back for the right context.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Relational (Postgres), user_features | You need to join a user's feature summary with order history, support tickets, or CRM data for offline analysis. | Wide array/vector columns hurt the query planner; keep the hot online copy in a KV store and use the relational copy for analytics/joins only. |
| Key-value / vector DB, item_embeddings and recommendation_cache | Access pattern is purely "get by id" - a vector for ANN search, or a cached list by cache_key - at very high scale. | No ad-hoc joins; if a request needs "recs joined with live inventory," that join happens in the ranking service's application code, not the store. |
7. Deep dives interviewers actually probe
How do you recommend anything for a brand-new user or item (cold start)?
For a new user with no interaction history, there's no personal embedding to search from, so the system falls back to content-based signals (category, brand, price band inferred from whatever context is available, like a signup survey or the landing page they arrived on) blended with globally popular items. For a new item with no purchase history, its embedding starts from item metadata (title, category, description) via a content encoder rather than co-purchase signal, and it's given an exploration boost - shown to a small slice of traffic even with a low predicted score - so the system can collect enough interaction data to learn a real embedding, often framed as a multi-armed bandit problem between exploiting known-good items and exploring new ones.
Why two-stage retrieval + ranking instead of scoring the entire catalog?
The ranking model is intentionally expensive per item - it looks at dozens of user/item/context features - which is affordable on 500 candidates but not on 350 million. An ANN index accepts a small, tunable recall loss (it might miss a few of the true best matches) in exchange for turning an O(catalog size) linear scan into an O(log n)-ish graph traversal, which is the only way candidate generation fits inside a low single-digit-millisecond slice of the 100ms budget.
// approximate nearest neighbor via HNSW: descend graph layers, greedy best-first search
function annSearch(queryVec, k):
candidates = entryPoint
for layer in topLayer downTo 0:
candidates = greedySearchLayer(queryVec, candidates, layer, ef=200)
return topK(candidates, k) // ~500 candidates out of 350,000,000 itemsHow do you stop recommendations from becoming a popularity filter bubble?
Left unchecked, a ranker trained on past clicks reinforces whatever was already popular, since popular items have the most training signal. Two common levers: reserve a small exploration slot per request (epsilon-greedy - with probability epsilon, swap in a lower-ranked or under-shown item instead of the top-scored one) so the model keeps getting feedback on items it's unsure about; and apply a diversity penalty at ranking time that down-weights an item's score the more times it's already been shown to this user this session, which both improves perceived variety and generates the exploration data needed to keep the model honest.
How stale can embeddings get before quality visibly degrades, and how do you pick retraining cadence?
Item embeddings drift slowly (a product's co-purchase neighborhood rarely changes hour to hour), so nightly batch retraining is usually fine for embeddings; user-level personalization is where staleness bites first, because a user's intent can shift within a single session - which is exactly why the online feature store exists, letting very recent clicks nudge ranking within the current session even though the underlying embeddings won't update until the next nightly run. The decision is a cost/benefit trade: continuous training needs constant validation infrastructure and risks silently shipping a regression, while batch retraining concentrates that risk into one reviewable nightly job with an offline metric gate.
How do you A/B test a new ranking model without risking revenue?
The rollout in the offline training sequence diagram is staged specifically to de-risk this: first shadow traffic (the new model scores real requests but its output is logged, not shown, so you can compare its rankings against production offline), then a small canary (5% of live traffic) with guardrail metrics watched in near-real-time (click-through rate, add-to-cart rate, and critically revenue-per-session, not just an offline NDCG number), and only then a ramp to 100% - with a held-out control group kept on the old model throughout so the comparison stays statistically clean and any regression can be rolled back by simply routing traffic back to the previous model registry version.
Post a Comment
Add