System design deep dive · HLD
Design a Feature Flag / Experimentation System (LaunchDarkly-style): full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for request-time evaluation and real-time config propagation, and an entity-relationship diagram for the data model - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
A strong system design answer starts by pinning down scope out loud. For a feature flag / experimentation platform, that means separating "evaluate a flag correctly and fast" from "propagate a change everywhere instantly" - two very different distributed-systems problems that live under one product.
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide almost everything downstream: why evaluation cannot touch the network, how big a config payload each SDK instance holds in memory, and how aggressively evaluation events must be sampled before they hit Kafka.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Active flags | 5,000 flags across the org | A full config snapshot is thousands of small rule objects - hundreds of KB, not GB. |
| Request volume | 2,000,000 requests/sec across the fleet | Baseline load driving evaluation volume. |
| Flag evaluations/sec | ~8 evaluations per request average | 2M × 8 = 16M evaluations/sec, ~100% done in-process (compute cost, not network cost). |
| Connected SDK instances | 50,000 service instances embed the SDK | Each needs the relevant flag config pushed on every change, not just at boot. |
| Config payload per client | ~300 KB per instance (flags relevant to that service) | Full broadcast ≈ 50,000 × 300KB ≈ 15 GB; a delta push after the first sync is orders of magnitude smaller. |
| Evaluation-event volume | Sample ~1% of evaluations, only for flags in an active experiment | 16M/sec × 1% = 160,000 events/sec × ~200 bytes ≈ 32 MB/sec sustained Kafka throughput, ≈ 2.7 TB/day. |
3. High-level design (HLD)
The HLD names the major components and the one-directional data flows between them: config authored once, config evaluated everywhere, evaluations logged asynchronously - without yet committing to specific regions, replica counts, or transport protocols.
What each box owns
Admin / API
Lets an operator create a flag, edit its default variant, and author ordered targeting rules ("enabled for segment X, or for 10% of traffic"). Writes go to the flag config store and nowhere else - the Admin/API never sits on the evaluation hot path.
Flag config store
The durable source of truth: flags and targeting_rules tables. Every write bumps a per-flag (or global) version number, which is the mechanism the distribution service uses to know something changed and to let a reconnecting client ask "what changed since version N?" instead of re-downloading everything.
Config distribution service
Holds long-lived connections (Server-Sent Events, or a websocket where bidirectional acks are wanted) to every connected SDK and pushes a config snapshot or delta the moment the version number changes. It is the piece that turns "an admin clicked save" into "every service reflects it within seconds," and it is why this system does not rely on polling or the next deploy.
Client SDK (embedded in each service)
Runs inside the host process. Holds the full relevant config in memory, and evaluates every flag with a deterministic hash-bucketing function - no network call per evaluation, ever. This is the component that makes 16 million evaluations/sec a non-issue: it's a hash and a comparison, not an RPC.
Evaluation event pipeline
The SDK batches "flag X evaluated as variant Y for user Z" events in memory and ships them asynchronously and off the hot path, sampled down (only for flags actually part of an active experiment) before landing in Kafka - so logging volume never scales 1:1 with evaluation volume.
Analytics / experiment store
Consumes the Kafka stream for offline analysis - which variant users saw, joined against downstream conversion events. The statistics engine that turns this into a significance result is a separate system and is explicitly out of scope here.
4. Detailed architecture diagram
The architecture diagram takes every HLD box and answers "how is this actually deployed and kept decoupled from the hot path?" - replica counts, the streaming transport, and exactly where the async event pipeline splits off so it can never add latency to an evaluation.
| Decision | Choice | Reasoning |
|---|---|---|
| Where evaluation runs | Fully in-process in the SDK, not a server round-trip | At 16M evaluations/sec, even a 1ms network round trip would add tens of thousands of CPU-seconds of waiting per second fleet-wide; local evaluation makes the cost a nanosecond-scale hash instead. |
| Config propagation transport | Streaming (SSE) over polling; websocket only where acks matter | Polling every few seconds either wastes requests when nothing changed or is too slow for a kill switch; a push-based stream delivers changes the moment the version bumps, at a fraction of the connection overhead of a websocket for a mostly server-to-client flow. |
| Bucketing approach | hash(user_id + flag_key) mod 100 | Concatenating the flag key makes bucket assignment independent per flag - a user in the top 10% for flag A is not correlated with their bucket for flag B - while staying deterministic and stateless (no lookup table to maintain). |
| Evaluation-event sampling | Sample + log only for flags in an active experiment | Logging every one of 16M evaluations/sec would make the event pipeline the bottleneck the flag system itself was built to avoid; sampling and filtering to "flags that matter for analysis" cuts volume by orders of magnitude with no loss of experiment validity. |
5. Sequence diagrams for the two critical flows
A sequence diagram is where an interviewer checks whether you actually understand what's synchronous versus fire-and-forget, and specifically whether evaluation ever waits on the network - it must not, in either normal operation or during a flag-change push.
5.1 Flag evaluation at request time
Steps 3-6 never leave the process - "read cached rules" is a field access on an object already resident in memory, not an RPC, which is why the arrow exists on the diagram purely to show call order rather than network hops. Step 7 is deliberately fire-and-forget: the SDK appends to an in-memory batch and returns immediately, so logging can never add latency to the response in step 8, and a full or slow batcher degrades to dropping events rather than blocking evaluation.
5.2 Real-time flag-change propagation (kill switch)
Step 4 is the whole point of the streaming transport: the admin's change reaches every live SDK within seconds, and step 5's atomic snapshot swap means no evaluation ever sees a half-updated config - it's either fully on v41 or fully on v42, never a mix of old and new rules for the same request. Steps 6-7 show the safety property that matters most for a kill switch: an SDK that loses its connection does not error or block, it keeps evaluating against its last cached version, and by design a kill-switch flag is configured to fail closed (off) on any doubt, while a cosmetic flag can be configured to fail open instead.
6. Entity-relationship (ER) diagram and schema
The data model has to answer three questions: how are ordered targeting rules attached to a flag, what makes bucketing deterministic and reproducible, and why the highest-volume table here doesn't really belong in a relational database at all.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Relational (Postgres/MySQL) for flags + targeting_rules | Low write volume (admin edits), need for strong consistency on version bumps and ordered-rule integrity. | Not built for the evaluation_events volume - don't be tempted to reuse the same instance for both. |
| Wide-column / columnar (Cassandra, ClickHouse, or a data-lake table) for evaluation_events | Extremely high append-only write volume, queried by time range and flag_id for offline analysis. | Point lookups by a single user_id across all time are expensive; partition by flag_id + time bucket to match the actual query pattern. |
7. Deep dives interviewers actually probe
How does consistent percentage bucketing avoid reshuffling users when a rollout grows from 10% to 50%?
Hash the user_id concatenated with the flag key into a stable number in [0, 99] - concatenating the flag key matters because it makes a user's bucket for flag A statistically independent of their bucket for flag B, instead of every rollout secretly correlating with the same underlying "lucky number." Rollout percentage is then just "which buckets count as in" - 0-9 for 10%, and growing that to 0-49 for 50% only ever adds buckets, never removes or renumbers the ones already in, so anyone already enabled stays enabled.
// Deterministic 0-99 bucket, independent per flag
function bucketFor(userId, flagKey) {
const h = murmurhash32(`${userId}:${flagKey}`);
return h % 100; // stable for this (user, flag) pair forever
}
function isEnabled(userId, flagKey, rolloutPercentage) {
return bucketFor(userId, flagKey) < rolloutPercentage; // monotonic expansion
}Why must evaluation happen fully in-process in the SDK rather than as a server round-trip?
At roughly 16 million evaluations per second fleet-wide, even a modest 1ms network round trip would add on the order of 16,000 CPU-seconds of pure waiting every second across the fleet, plus make every flag check a new point of failure and a load-balancer hop. The SDK sidesteps this entirely by caching the full config relevant to its service locally and treating evaluation as a hash lookup and a comparison - correctness is maintained by keeping that local cache continuously synced via the streaming distribution channel, not by asking a server on every call.
What happens when a client's connection to the config-distribution service drops?
The SDK keeps evaluating against its last-known-good cached config rather than blocking the calling request or throwing - a disconnected SDK is still a correctly-functioning SDK, just a slightly stale one. Whether "stale" should mean "fail open" or "fail closed" is deliberately a per-flag decision rather than a global one: a kill-switch flag (turning off a broken feature) should default to off/closed if evaluation ever can't confirm the latest state, while a low-risk UI-tweak flag can safely fail open and keep showing its last-seen variant, because the cost of staleness is asymmetric across flag types.
How do you keep evaluation-event logging from becoming a bottleneck or a huge cost at billions of evaluations a day?
Three levers, applied together: the SDK batches events in memory and ships them on an interval or size threshold rather than one network call per evaluation; only a sampled fraction of evaluations are logged (the capacity table above uses ~1%, tunable per experiment's required statistical power); and, more importantly, events are only generated at all for flags actually attached to an active experiment - a boolean ops kill-switch flag with no experiment behind it produces zero logging overhead, because there's nothing to analyze.
How would you support targeting on complex custom attributes without turning the rule engine into an unbounded query language?
Constrain segment_condition to a fixed, small set of operators over known attribute types - equals/in for strings and enums, greater-than/less-than/between for numbers and dates, contains for sets - rather than accepting an arbitrary expression or embedded scripting language. "Enabled for enterprise-tier accounts created after date X in region EU" becomes a small AND of three typed comparisons the SDK can evaluate in microseconds and the config UI can render as a form, not a query planner - the same discipline that keeps evaluation fast also keeps the targeting rules auditable and impossible to accidentally turn into a denial-of-service vector.
Post a Comment
Add