System design deep dive · HLD
Design Instagram Stories: ephemeral content, full high-level design.
Requirements, back-of-envelope capacity estimation for a celebrity-scale fan-out read problem, a high-level design diagram, a deeper deployment architecture, sequence diagrams for uploading and viewing a story, and an entity-relationship diagram built around a hard 24-hour TTL - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
Stories are defined by one property that changes everything downstream: content is meant to disappear. That single constraint - a hard 24-hour lifetime - drives the data model, the caching strategy, and the deletion pipeline more than any other requirement.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
Uploads are cheap and rare compared to views. The number that decides almost every architecture choice below is the view:upload ratio - it is what rules out fanning writes out to followers and forces a pull-based, cache-first read path.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Daily active users | 500 million DAU | Baseline for every ratio below |
| Stories posted (writes) | ~20% of DAU post a story/day | 100M stories/day → ~1,160 writes/sec avg, ~3,500/sec peak |
| Views per story (skewed) | ~150 views/story average; top creators reach millions within hours | 100M stories × 150 ≈ 15 billion view events/day |
| View-event QPS (the hot path) | 15B events / 86,400s, ~3x multiplier at peak hours | ~174,000/sec avg, ~500,000+/sec peak - a 150:1+ read:write ratio, spiking far higher per hot story |
| Blended media size | 70% photo (~200KB compressed) + 30% video (~5MB, 15s clip) | 0.7×200KB + 0.3×5MB ≈ 1.64MB average per story |
| Rolling storage window | 100M stories/day × 1.64MB, retained ~27h (24h visible + safety buffer) | ~164TB/day ingested → ~185TB resident in the ephemeral tier at steady state |
| CDN egress bandwidth | 15B views/day × ~800KB avg delivered (thumbnails + adaptive video chunks, not full re-downloads) | ~12PB/day served at the edge; origin fetches only cache misses, well under 5% of that |
3. High-level design (HLD)
The HLD splits cleanly into a write path (post a story), a read path (load the ring, fetch media), and two side pipelines that must never block either path: view tracking and TTL expiry.
What each box owns
Client & API gateway/edge
The gateway terminates TLS, authenticates the request, and applies per-user upload rate limits before anything reaches the write path. It also fetches media directly from the CDN rather than proxying it through the app tier - the dashed arc in the diagram shows media flowing straight from object storage back to the client, never through the ring assembly service.
Upload service, transcode pipeline & object storage + CDN
The upload service accepts the raw photo/video and hands it to the transcode pipeline, which resizes photos and re-encodes video into adaptive-bitrate chunks plus a thumbnail. The processed media lands in object storage behind a CDN, which is what absorbs the celebrity fan-out - the file is written once and served from edge caches for every subsequent view.
Metadata service & TTL expiry sweep
The metadata service is the source of truth for each story's expires_at, view count, and active flag. The expiry sweep runs independently (native TTL or a scheduled job, see the deep dive) and only ever deletes rows that are already invisible to reads - the read path filters on expires_at regardless of whether physical deletion has happened yet.
Ring assembly service & cache
On every app open, this service resolves "which of the accounts I follow have an active story right now, and which have I not seen yet." It checks the Redis cache first; a hit returns in low single-digit milliseconds, and only a miss touches the metadata database, which is what keeps the p99 latency target achievable at 500M DAU.
View-tracking service, Kafka & view-count store
The client publishes a view event the moment a story starts playing, but never waits for an acknowledgment - it's a fire-and-forget call into the view-tracking service, which drops the event onto a Kafka topic. A stream processor consumes the topic and updates the view count and per-viewer "who viewed" list asynchronously, decoupled entirely from the read-hot ring path.
4. Detailed architecture diagram
The architecture diagram answers the one question that decides whether this design survives a celebrity posting a story: how do you serve tens of millions of reads against one piece of media without hammering a single database row or fanning writes out to every follower?
| Decision | Choice | Reasoning |
|---|---|---|
| Fan-out model | Pull (cache + CDN), not push to every follower's feed | A celebrity story can be viewed tens of millions of times in hours; writing a copy into every follower's feed at post time doesn't scale, while one cached CDN object serves unlimited readers. |
| Metadata sharding key | Hash of story owner's user_id | Every ring query is "give me this followee's active stories" - a natural per-owner lookup; the ring cache absorbs the scatter-gather across a viewer's many followees. |
| TTL deletion mechanism | Native TTL attribute, backed by a sweep worker | Native TTL is free and low-maintenance for bulk cleanup; the sweep worker exists for cases where TTL's built-in deletion lag is unacceptable (see deep dive 1). |
| CDN caching strategy | Immutable, content-addressed URLs with long-lived cache-control | Media never changes after upload, so edge caches can hold it aggressively without any invalidation logic - only new uploads get new URLs. |
5. Sequence diagrams for the two critical flows
The upload flow is dominated by processing steps that must finish before the story becomes visible; the view flow is dominated by a cache check that must resolve in milliseconds because it runs 150 times more often than an upload.
5.1 Post a story (write path)
Step 3 is the slowest part of the whole flow and runs entirely inside the transcode pipeline with no network round trip drawn, which is why upload latency is dominated by video re-encoding, not by any database write. Step 8 is deliberately dashed and fire-and-forget: invalidating every follower's cached ring is an optimization for freshness, not a correctness requirement, so the client's 201 response in step 9 never waits on it.
5.2 View the story ring (read path, with cache miss)
On a cache hit, steps 3-6 collapse into a single Redis read and the ring returns in a few milliseconds - the path that must hit the <200ms p99 target. Step 8 goes straight from client to the CDN, bypassing the ring assembly service entirely, which is exactly what lets one celebrity story absorb millions of views without touching the metadata database again. Step 9 never blocks playback. The TTL sweep that eventually deletes this same story runs completely independently of this request path - on its own schedule, against the metadata database directly - so an in-flight ring load is never affected by expiry timing, and a story disappears from new ring loads immediately once expires_at passes even if its row has not been physically deleted yet.
6. Entity-relationship (ER) diagram and schema
The data model has to answer three questions: what marks a story as visible right now, how are views recorded without write-amplifying a single hot story row, and how is the 24-hour deletion actually implemented at the storage layer.
Key modeling decisions
expires_at > now(); the same column doubles as the native TTL attribute if the storage engine supports it.| Storage choice | Use when | Watch out for |
|---|---|---|
| Sharded relational (Postgres, by user_id) | You want a strict uniqueness constraint on (story_id, viewer_id) for view dedupe and simple transactional writes on upload. | No native per-row TTL, so it needs the separate expiry_jobs table and a sweep worker to actually reclaim storage. |
| NoSQL wide-column / KV (DynamoDB, Cassandra) | Access pattern is purely key-based ("stories where user_id=X and expires_at>now") at very high write/read throughput. | Native TTL deletion has no fixed SLA (DynamoDB can lag up to ~48 minutes past expires_at), so the read path - not the physical delete - is what must enforce visibility. |
7. Deep dives interviewers actually probe
How exactly does the 24-hour TTL get enforced?
Two mechanisms, often used together. Native TTL (e.g. DynamoDB's TTL attribute) is free and requires no compute, but its background reaper only guarantees eventual deletion - AWS documents deletion lag of up to roughly 48 minutes past the timestamp, so it cannot be trusted for precise, instant removal. An application-level scheduled sweep job queries for expired rows in batches and deletes them on a tight, predictable schedule, at the cost of running dedicated compute continuously. The key insight is that neither mechanism needs to be exact, because the read path enforces visibility independently: every ring query filters WHERE expires_at > now(), so a row that hasn't been physically deleted yet simply never appears to a viewer.
// DynamoDB item - ttl is the native expiry attribute (epoch seconds)
{
"story_id": "8f2c...",
"user_id": "1029384",
"expires_at": 1783650000, // application-level filter uses this
"ttl": 1783650000 // same value, reaper deletes eventually
}
// Read path never trusts physical deletion timing:
// SELECT ... WHERE user_id IN (:following) AND expires_at > :nowHow do you handle a celebrity story getting millions of views in hours?
By never fanning the write out in the first place. A push model - writing a copy of the story reference into every follower's feed at post time - turns one upload into tens of millions of writes, and a celebrity gaining or losing followers mid-story makes it worse. The pull model used here stores the media once in object storage behind a CDN and lets every viewer's read resolve against the same cached object; the metadata row for a hot story is read far more often than a normal story, but it's a read, not a write, so it scales by adding cache and edge capacity rather than by fanning out writes. The only extra work for a hot story is that its ring-cache entry and CDN edge cache get pre-warmed and given a longer TTL, since it's known in advance to be disproportionately hot.
How is the story ring ordered, and how is "seen" tracked cheaply?
Ordering is unseen-first, then by a recency/affinity score (recent post time, weighted by how often the viewer engages with that account). The naive way to track "seen" - one row per (viewer, story) pair - creates a write on every single scroll across hundreds of millions of users. Instead, store a single lightweight cursor per (viewer, followee) pair: the highest story ID or timestamp that viewer has already seen from that account. Because a user's own stories are strictly ordered by creation time, comparing the cursor to each candidate story's ID/timestamp is enough to classify it as seen or unseen, with one small write per followee interaction instead of one per story per scroll.
// one row per (viewer, followee), not per (viewer, story)
cursor = getSeenCursor(viewer_id, followee_id) // { last_seen_story_id }
isUnseen = story.id > cursor.last_seen_story_id
// updating the cursor to the latest story_id covers every older
// story from that followee in a single writeWhat durability guarantees does the media need during the 24h window?
The metadata row is the actual source of truth for visibility - once expires_at passes or is_active flips false, the story stops appearing in any new ring load, regardless of whether the media file itself has been deleted. The object storage lifecycle rule intentionally deletes the underlying file later (around the 25-26h mark), so a client who already loaded the ring a few seconds before expiry and is mid-playback doesn't hit a broken CDN URL. This is also why the consistency requirements differ per field: view counts can lag by seconds to minutes with zero user-visible harm because they're purely informational, but expiry timing cannot drift by more than a small, bounded amount, because showing an expired story past its promised lifetime is a product and privacy violation, not just a staleness issue.
How would you extend this to support "highlights" that persist past 24h?
Not by clearing expires_at on the original row - that row is already wired into the ephemeral pipeline (native TTL, lifecycle-tagged media, "unseen-first" ring logic), and mixing permanent rows into it re-introduces every problem the TTL design solved. Instead, saving a story to highlights performs a copy-on-save: a new row is written into a separate highlights / highlight_items table with no expires_at, referencing either the same media object promoted to a "retained" storage class (so the ephemeral lifecycle rule no longer deletes it) or a copy of the object into a permanent bucket. The core ephemeral pipeline - upload, transcode, 24h expiry - stays completely unchanged; highlights become an entirely additive read path layered on top.
Post a Comment
Add