System design deep dive · HLD
Design Pastebin: full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for paste creation-with-expiry and retrieval-with-view-tracking, and an entity-relationship diagram for the data model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
A strong system design answer starts by pinning down scope out loud. For Pastebin, that means separating this from a URL shortener up front: the interesting problems here are content size variance, expiry-driven storage lifecycle, and view tracking - not redirect mechanics or hash collisions.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide the inline-vs-blob-storage size threshold, how aggressively expired data can be reclaimed, and whether the metadata table alone (without content) is small enough to keep entirely cache-warm.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| New pastes (writes) | 8 million/day | ~93 writes/sec average, ~800 writes/sec peak |
| Paste retrievals (reads) | 30:1 read:write ratio | ~2,800 reads/sec average, ~24,000 reads/sec peak |
| Paste size distribution | 90% under 10KB, 9% under 512KB, 1% up to a few MB | Median paste is small enough to store inline with metadata; the 1% tail drives the blob-store need |
| Storage growth | ~8M/day × ~8KB average (blended) × 90-day median retention | ~5.7 TB live at any time before expiry reclamation, growing more slowly than raw write volume thanks to expiry |
| Cache size for hot pastes | ~5% of daily pastes account for ~70% of reads (viral shares) | ~400K hot pastes/day × ~8KB ≈ 3.2 GB - comfortably fits a Redis/CDN cache tier |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flow between them, without committing yet to specific infrastructure, regions, or replica counts - that level of detail belongs in the architecture diagram in the next section.
What each box owns
Create service (write path)
Accepts the paste content, visibility, and expiry choice; requests a globally unique short ID; decides inline-vs-blob storage purely on content size against the threshold (512KB in this design); writes metadata (and inline content, if small) to the metadata store, or writes large content to the blob store first and records only a reference in metadata. Visibility is enforced at write time by generating an unguessable ID for unlisted/private pastes rather than a short sequential one, since "unlisted" security depends entirely on the ID being unguessable.
Retrieve service (read path)
Looks up metadata by ID first (cache, then DB), checks expiry and visibility/auth before returning anything, then either returns the inline content directly or fetches the referenced blob. It publishes a "view happened" event to the view-count queue and returns without waiting for that event to be processed, exactly like the click-tracking pattern in a URL shortener - view analytics must never add latency to a paste being read.
ID generator
For public pastes, a short Base62 ID derived from a Snowflake-style unique numeric ID (cheap to guess, but that's fine - public pastes are meant to be discoverable). For unlisted/private pastes, a longer cryptographically random ID (128+ bits of entropy) is generated instead, since "unlisted" is a security property that depends entirely on the ID space being too large to enumerate or guess.
Metadata cache/store, blob store, view-count queue, and expiry reaper
The metadata store holds every paste's id, owner, visibility, expiry, and either inline content (small pastes) or a blob reference (large ones) - kept small and cache-friendly since it's on every read's critical path. The blob store holds only large paste bodies, addressed by content hash, and is never consulted for the 90% of pastes that fit inline. The view-count queue batches increments asynchronously rather than writing on every single read. The expiry reaper runs as a background job that deletes expired metadata rows and their associated blobs, reclaiming storage without needing every read to double as a cleanup check.
4. Detailed architecture diagram
The architecture diagram takes every HLD box and answers "how is this actually deployed?" - replica counts, sharding, regions, and the specific technology choice, which is what an interviewer is checking for once they've accepted the high-level shape.
| Decision | Choice | Reasoning |
|---|---|---|
| Inline vs blob threshold | 512KB, chosen from the size-distribution estimate | Keeps 90%+ of pastes servable from a single cache/DB lookup with no second blob-store round-trip, while still handling multi-megabyte log dumps without bloating the hot metadata table. |
| Blob expiry enforcement | Object storage lifecycle policy, not an application-level delete job alone | Belt-and-suspenders: the storage layer itself guarantees reclamation even if the application-level expiry reaper has a bug or an outage. |
| Unlisted/private paste ID entropy | 128+ bits of cryptographic randomness, distinct from the short public-paste ID scheme | "Unlisted" is a security property enforced purely by the ID being unguessable; a short, dense ID space would make brute-force enumeration of private content plausible. |
| Malware/secret scanning | Asynchronous, post-write, not blocking paste creation | Scanning for leaked credentials or malicious payloads must not add latency to every paste creation; instead it flags/removes content shortly after write, accepting a brief window of exposure as the trade-off. |
5. Sequence diagrams for the two critical flows
A sequence diagram is where an interviewer checks whether you actually understand call order, what is synchronous versus fire-and-forget, and exactly where the inline-vs-blob branch happens.
5.1 Create paste with expiry and visibility (write path)
Step 4's branch is the single most important decision point in the whole write path: a small paste would instead write its content directly into the metadata row in step 6 and skip the blob store entirely, saving a network round-trip for the 90%+ common case. Step 2 requests high-entropy ID generation specifically because this paste is unlisted - had visibility been public, a cheaper, shorter Base62-encoded sequential ID would be requested instead, since guessability isn't a concern for content the owner wants discoverable.
5.2 Retrieve paste with view-count tracking (read path)
Step 4's expiry and visibility check happens before ever touching the blob store, so an expired or unauthorized request is rejected cheaply without an unnecessary storage read. Step 6's fire-and-forget view event is the same pattern as a URL shortener's click tracking - view-count accuracy is valuable to the paste's owner but must never gate whether the reader actually sees the content, so the response in step 7 does not wait for it.
6. Entity-relationship (ER) diagram and schema
The data model has to answer three questions: where does small content live versus large content, how is view volume recorded without write-amplifying the hot metadata row, and how is the inline/blob split actually represented.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Relational (Postgres/MySQL), sharded by hash(paste_id) | You want inline content and metadata in one row for a single-lookup read, plus straightforward joins for an owner's paste-list dashboard. | Very large inline_content values (near the 512KB threshold) bloat row size and index locality; the threshold must be enforced strictly. |
| Key-value with native TTL (DynamoDB) | Expiry is a first-class requirement and you want the storage layer itself to reclaim expired items without a custom reaper job. | Content-hash-based blob dedup and cross-paste aggregate queries (e.g. "most-viewed public pastes today") are harder to express than in a relational model. |
7. Deep dives interviewers actually probe
How do you decide the inline-vs-blob storage threshold, and does it ever change?
The 512KB threshold is chosen from the size-distribution data (capacity section): it's set high enough to keep the vast majority of pastes as a single fast lookup, but low enough that the metadata table's row size stays predictable and cache-friendly. It is intentionally a config value, not a hardcoded constant, since a platform's actual paste-size distribution can shift over time (e.g. more log-dump-style pastes as it gains adoption in an ops-tooling context) and the threshold should be re-tuned from real percentile data periodically rather than fixed at launch.
How is "unlisted" actually enforced - is it real security or just obscurity?
It is explicitly security-through-obscurity, and the design should say so plainly rather than imply otherwise: unlisted pastes rely entirely on the ID being computationally infeasible to guess (128+ bits of randomness, never listed or indexed anywhere, never appearing in a sitemap or search-engine-crawlable page). This is materially weaker than "private" (which requires actual authentication) and that trade-off should be surfaced to users - an unlisted paste's URL, if it ever leaks via a referrer header, browser history sync, or being pasted into a public chat, becomes effectively public.
// Unlisted/private IDs use real entropy; public IDs can be short and guessable by design
String unlistedId = base62Encode(secureRandomBytes(16)); // ~128 bits
String publicId = base62Encode(snowflakeId()); // dense, sequential-ish, fine to guessHow does expiry actually get enforced end-to-end, not just "hidden from listing"?
Two independent layers, deliberately redundant: the retrieve service checks expires_at against the current time on every read regardless of cache state (so even a stale cache entry can't serve expired content past its expiry instant), and a background reaper plus the blob store's own lifecycle policy physically delete the underlying data afterward. The read-time check is what guarantees expired content is genuinely inaccessible immediately, independent of how promptly the asynchronous cleanup job gets around to reclaiming its storage.
How do you handle abuse - malware droppers, leaked credentials, or C2 payloads pasted as "plain text"?
An asynchronous post-write scanner (architecture diagram) checks new pastes against known-secret patterns (API key formats, private key headers) and malware-signature/URL-reputation databases, flagging or removing matches shortly after creation - deliberately not blocking creation synchronously, since scanning against large signature databases is too slow to sit on the write's critical path. This accepts a short exposure window as the trade-off for keeping paste creation fast for the overwhelming majority of legitimate pastes.
What is the single biggest bottleneck as this scales 10x?
Not the read path - cache hit rate and the CDN layer keep that cheap and horizontally scalable. The real bottleneck becomes blob-store write and reclaim volume for the large-paste tail: at 10x scale, even a small percentage of multi-megabyte pastes adds up to serious storage churn, and expiry reclamation has to keep pace with creation or storage grows unbounded. This is solved by making sure the expiry reaper (and the object store's native lifecycle policy) scales independently as a background batch process, rather than trying to squeeze cleanup into the synchronous request path anywhere.
Post a Comment
Add