System design deep dive · HLD
Design YouTube: full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for asynchronous upload/transcoding and for view-counting at scale, and an entity-relationship diagram for videos, views, channels, and comments - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
YouTube's hardest problem is not any single feature - it's that upload, playback, view counting, comments, and recommendations all have wildly different consistency and latency needs, and mixing them up in one datastore would break all of them at once.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide whether view counting can ever be a synchronous database increment (it cannot), how large the transcoding fleet needs to be, and how comments must be partitioned to avoid hotspotting under a viral video.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| New uploads | 500 hours of video uploaded per minute | ~720,000 hours/day ≈ ~30,000 concurrent transcoding jobs assuming ~1hr avg processing time per hour of source |
| Video views | ~5 billion views/day globally | ~58,000 views/sec average, ~180,000 views/sec peak (prime-time regions overlapping) |
| View-count increments | Every view is a counter write, batched | Naive per-view DB writes at 180K/s would saturate any single database; must be buffered and aggregated |
| Comments posted | ~0.2% of views produce a comment | ~58,000 × 0.002 ≈ 116 comment writes/sec average - far lighter than the view path |
| Storage growth | 720,000 hours/day × ~1.5GB/hr average across the rendition ladder | ~1.08 PB/day of new encoded video, all going to durable, replicated object storage |
3. High-level design (HLD)
The HLD separates four independent paths that only share the video's metadata record: the upload/transcode pipeline, the watch/streaming path, the view-counting pipeline, and comments - each scaled and consistency-tuned for its own workload.
What each box owns
Upload service & transcode workers
The upload service accepts chunked, resumable uploads so a spotty connection doesn't force a creator to restart a multi-GB file from zero. Once fully received, the raw file is handed to a queue-driven pool of transcode workers that produce the rendition ladder (multiple resolutions/codecs) and thumbnails asynchronously - the creator gets an immediate "processing" confirmation, not a synchronous wait for encoding to finish.
Watch service & video metadata
Resolves a video ID to its manifest (available renditions, CDN URLs) and current metadata (title, view count snapshot, channel). This is the highest-QPS read path in the whole system and is backed by a heavily cached, sharded metadata store - it deliberately never touches the view-counting pipeline synchronously.
View counter aggregator
Every watch emits a lightweight view event to a Kafka stream rather than incrementing a database row directly. A windowed aggregator consumes the stream, applies bot/fraud filtering heuristics (e.g. requiring a minimum watch duration, deduplicating rapid repeat views from the same session), and periodically flushes batched increments to the metadata store - trading exactness for the ability to absorb 180,000 events/sec without falling over.
Comments service & recommender
Comments live in their own sharded store (sharded by video ID) so a comment storm under one viral video can't degrade writes for unrelated videos. The recommender consumes watch history and video metadata offline/near-real-time to produce a personalized ranked feed per viewer, served from a precomputed cache rather than computed synchronously on every home-page load.
4. Detailed architecture diagram
The architecture diagram shows how the view-counting pipeline is deliberately decoupled and buffered, how transcoding is sharded across an elastic worker pool, and how comments are sharded to survive a single video going viral.
| Decision | Choice | Reasoning |
|---|---|---|
| View counting | Stream + windowed aggregation, batched flush | 180,000 events/sec of synchronous per-row increments would saturate any relational database; batching absorbs bursts and tolerates a display lag of a few seconds. |
| Metadata sharding key | Hash of video_id | Every watch, comment, and view-count update is keyed by video_id, so sharding on it keeps almost all queries single-shard. |
| Comments isolation | Separate sharded store from video metadata | A comment storm under a viral video must not compete for capacity with the metadata store that every playback request depends on. |
| Transcode worker scaling | Elastic pool, priority queue by channel size/verification | Upload volume is bursty; a large creator's time-sensitive upload shouldn't wait behind a backlog of low-priority re-encodes. |
5. Sequence diagrams for the two critical flows
The upload flow shows how transcoding is fully decoupled from the client response; the view-counting flow shows exactly why a "view" is never a synchronous database write.
5.1 Video upload & asynchronous transcoding
Step 2 returns immediately after the raw bytes are durably stored, well before encoding starts - the creator's upload experience is never gated on transcoding time. Steps 4-5 can take anywhere from minutes to hours depending on queue depth and source length, which is exactly why the response in step 2 is a 202 Accepted with a polling/webhook status rather than a synchronous "video ready" response.
5.2 View counting at scale
Step 3 is fire-and-forget from the player's perspective and only fires after a minimum watch duration threshold - a 2-second drive-by click never counts as a view. Steps 4-6 batch potentially thousands of individual view events into a single UPDATE per window per video, which is what makes 180,000 events/sec survivable; the visible view count on screen (step 7) is therefore always a few seconds stale by design, not a bug.
6. Entity-relationship (ER) diagram and schema
The schema has to answer: how is a channel's video list retrieved without a scan, how are views recorded without write-amplifying the hot videos row, and how are threaded comment replies modeled.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Relational (Postgres/MySQL) for channels & videos | Comparatively low write volume, benefits from strong constraints and channel-video joins for the studio dashboard. | Not designed for the view/comment write-fan-out of a viral video. |
| Wide-column/document (Cassandra/DynamoDB) for views & comments | Access pattern is always "this video's events" at extreme, unpredictable write volume. | Cross-video aggregate queries (trending, top creators) need a separate batch/OLAP pipeline. |
7. Deep dives interviewers actually probe
Why can't a "view" ever be a direct database increment?
At ~180,000 views/sec peak, a naive UPDATE videos SET view_count = view_count + 1 per view would create row-lock contention concentrated on the handful of currently-viral videos, turning them into hot spots that slow down for everyone watching them at the exact moment they're most popular. Routing views through a stream and aggregating in time windows converts millions of tiny writes into a much smaller number of batched updates, and naturally smooths out bursts.
How do you stop bots and refresh-spam from inflating view counts?
The aggregator applies heuristics before a view counts at all: a minimum watch duration threshold (e.g. 30 seconds or a meaningful percentage of the video), session-level deduplication (the same viewer re-watching within a short window counts once), and signals like IP/device velocity and known bot user agents. None of this needs to be perfectly accurate in real time - it can be refined by an offline batch job that periodically reconciles and corrects counts.
How does a comment section survive a video going viral overnight?
Because comments are sharded by video_id, a flood of comments under one viral video only load-tests that video's shard, not the whole comments fleet. Pagination is cursor-based (by comment ID or timestamp, not offset) so deep pagination on a comment thread with millions of entries doesn't degrade into an expensive scan.
How is the personalized recommendation feed generated without recomputing it on every page load?
Recommendations are precomputed offline/near-real-time by a batch or streaming ML pipeline that consumes watch history and produces a ranked candidate list per viewer, cached and refreshed periodically (e.g. every few minutes to hours). The home page reads this precomputed list rather than running a ranking model synchronously per request, which would be far too slow and expensive at this read volume.
What is the single biggest bottleneck as this scales 10x?
Not playback - CDN edge and read replicas scale that horizontally. The real bottleneck becomes the view-counting aggregation layer during a global viral event (a single video watched by tens of millions within hours): the windowed aggregator for that one video_id's partition can become a hotspot even with batching. The fix is finer sub-partitioning of hot video_ids across multiple aggregator instances with a final merge step, not simply widening the batch window.
Post a Comment
Add