System design deep dive · HLD
Design a Content Moderation system: full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for automated ML classification and human-review escalation, and an entity-relationship diagram for the data model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
A strong system design answer starts by pinning down scope out loud. For content moderation, that means separating "flag it fast with a model" from "get it right when the model is unsure" - the human review and appeal paths are what most interviewers actually want to hear reasoned through.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide how many GPU/CPU inference workers are needed, how big the human review team must be, and whether review queue storage needs to be sharded.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Daily uploads | 50 million items/day | ~580 items/sec average, ~4,000 items/sec peak |
| ML classification cost | ~40ms/item on a shared inference fleet | ~4,000 × 0.04s ≈ 160 concurrent inference slots needed at peak |
| Escalation rate | ~2% of items score in the "borderline" band | ~1M items/day ≈ 12/sec average routed to human review |
| Reviewer throughput | ~90 seconds average review time/item, 6.5 productive hours/shift | ~1M/day ÷ (3,600×6.5÷90) ≈ ~3,850 reviewer-shifts/day globally, spread across follow-the-sun shifts |
| Storage per content_item metadata | ~1 KB (scores, hashes, decision, policy version) | 50M/day × 1KB × 2yr retention ≈ 36 TB - sharded relational or wide-column store |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flow between them, without committing yet to specific infrastructure, regions, or replica counts - that level of detail belongs in the architecture diagram in the next section.
What each box owns
Upload service
Accepts the content, stores the raw media in blob storage, computes a perceptual hash (for known-bad-content matching) and a content_item row, then emits an event rather than calling the classifier synchronously - so a slow or backed-up classification pipeline never adds latency to the upload itself, and content can go live immediately under a "provisionally visible" state pending classification, or held pending classification depending on the product's risk tolerance for that surface.
Classification service
Consumes the upload event, runs the appropriate model(s) per media type (a CNN-based image classifier, a text transformer for captions/comments, a perceptual-hash lookup against known-violation hash sets), and produces a per-category score vector. It is deliberately stateless and horizontally scaled by adding inference workers, since ML inference is the component most likely to need GPU capacity that scales independently of the rest of the system.
Decision engine
Applies policy thresholds per category to the score vector: below the low threshold the content is allowed automatically; above the high threshold it is automatically removed (for severe, high-confidence categories); in between, it is enqueued for human review. Thresholds are versioned and stored alongside the decision so that a later policy change never silently reinterprets historical decisions.
Review queue, moderator UI, and the content database
The review queue is priority-ordered (severity score, content age, reporter count) rather than strict FIFO, so a rapidly-spreading harmful item is reviewed before an old low-severity one. The moderator UI presents the content, the model's scores, and relevant context, and writes the human verdict back to the content database - which is the single source of truth an appeal later reads from.
4. Detailed architecture diagram
The architecture diagram takes every HLD box and answers "how is this actually deployed?" - replica counts, GPU pools, regions, and the specific technology choice, which is what an interviewer is checking for once they've accepted the high-level shape.
| Decision | Choice | Reasoning |
|---|---|---|
| Human review staffing model | Follow-the-sun desks across regions, not a single 24/7 site | Keeps queue latency bounded around the clock without requiring any single site to staff unpopular overnight shifts. |
| Review queue ordering | Priority score (severity × reach × age), not FIFO | A viral harmful post reaching thousands of views must be reviewed before an old low-severity item sitting in the queue, even if it arrived later. |
| Known-bad hash matching | Perceptual hash lookup runs before the ML model, not after | A cheap hash-set lookup against previously confirmed violations (e.g. re-uploads of already-removed content) can auto-remove instantly without waiting on GPU inference capacity. |
| Feedback loop isolation | Human verdicts feed model retraining asynchronously via a separate topic | Retraining pipelines must never share load or failure domains with the synchronous classification path that gates content visibility. |
5. Sequence diagrams for the two critical flows
A sequence diagram is where an interviewer checks whether you actually understand call order, what is synchronous versus fire-and-forget, and exactly which threshold decides between auto-action and human escalation.
5.1 Automated ML classification on upload
Steps 1 and 8 are drawn as fire-and-forget so that classification latency never blocks the upload response the user already received in step 1's originating request. Step 5's policy thresholds are versioned (v14) and stamped onto the content_item record so that a later threshold tuning never retroactively reinterprets a decision that already shipped - important for appeal fairness, since an appeal must be judged against the policy in force at the time.
5.2 Escalation to human review and appeal override
Step 7 is the detail interviewers most want named explicitly: an appeal must never be routed back to the same reviewer who made the original call, both to avoid anchoring bias and to satisfy the auditability requirement that the review process itself is defensible. Step 8's rationale is written to the same immutable decision log as the original verdict, so the full history - model score, first human verdict, appeal verdict - is reconstructable for compliance review.
6. Entity-relationship (ER) diagram and schema
The data model has to answer three questions: where do ML scores and human verdicts both live without overwriting each other, how is the review queue ordered without a full table scan, and how does an appeal reference the original decision it is contesting.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Relational (Postgres), sharded by content_id | You need transactional "read flag, write verdict, update content status" consistency and straightforward joins for the moderator UI. | Cross-shard aggregate queries (e.g. global category trend dashboards) need a separate analytics replica. |
| Wide-column (Cassandra) for moderation_flags | Write volume is dominated by ML flags at 50M/day scale and access is mostly append + read-by-content_id. | Priority-ordered queue semantics and cross-flag joins are awkward; pair with a dedicated queue service. |
7. Deep dives interviewers actually probe
How do you set the auto-remove vs human-review threshold per category?
Thresholds are asymmetric per category, not a single global cutoff. Severe, hard-to-misclassify categories (known-hash CSAM matches, explicit violence with high model confidence) use a low bar for auto-removal because the cost of a false negative is unacceptable. Nuanced categories (satire vs hate speech, artistic nudity vs explicit content) use a wide borderline band that routes to human review even at moderate confidence, because the cost of a wrongful auto-removal (chilling legitimate speech, reviewer backlash) outweighs the cost of a short review delay.
How do you prevent reviewer bias and inconsistency at scale?
Every decision is logged with the reviewer's identity and the exact content/scores they saw (moderation_flags), enabling periodic inter-rater agreement audits - sampling the same borderline items to multiple reviewers and measuring disagreement rate. Appeals are routed to a reviewer who did not make the original call (sequence 5.2, step 7), and reviewers with abnormal override rates or agreement scores are flagged for retraining, not silently left to keep deciding cases their way.
How does the appeal path avoid becoming a second attack surface?
Rate limiting on appeal submission per user, and a rule that an appeal on a previously-appealed decision does not get infinite additional appeals (typically capped at one escalation tier, e.g. reviewer to senior-reviewer to policy team) - otherwise a bad actor could indefinitely delay enforcement on genuinely violating content by repeatedly appealing, which the design explicitly guards against by making the second-level review binding.
What happens to content while it's waiting in the review queue?
Depends on the confidence band and the surface: content in the "likely fine, mildly ambiguous" band typically stays visible while queued (optimizing for user experience, since most of it will be confirmed fine); content that scored high enough to be concerning but not high enough for auto-removal is often held/limited-distribution while awaiting review, trading some legitimate content's temporary reduced reach for meaningfully lower harm exposure during the review window.
How does the human-verdict feedback loop avoid model drift or gaming?
Human verdicts become training labels via the async retrain feedback topic (architecture diagram), but naively retraining on 100% of verdicts can bias the model toward whatever the human reviewers are currently seeing (which is itself already filtered by the model's own thresholds - a feedback loop that can amplify blind spots). The mitigation is a small, randomly sampled "exploration" slice of traffic that bypasses the normal threshold and goes to human review regardless of model score, purely to keep the training label distribution unbiased.
What is the single biggest bottleneck as this scales 10x?
Not ML inference - that scales by adding GPU workers. The real constraint becomes human reviewer capacity, since the 2% escalation rate at 10x upload volume means 10x the review queue depth, and reviewer headcount does not scale as elastically as compute. That is addressed by continuously narrowing the borderline band as the model's precision improves (fewer items truly need a human), and by triaging within the queue so reviewer time is spent on the highest-severity, highest-reach items first rather than processing strictly in arrival order.
Post a Comment
Add