System design deep dive · HLD
Design an E-commerce Cart & Checkout System: full high-level design.
Requirements, back-of-envelope capacity estimation, a high-level design diagram, a deeper deployment architecture diagram, sequence diagrams for cart-to-inventory-reservation and for the order/payment/shipping saga with compensation, and an entity-relationship diagram for carts, orders, inventory, and payments - with the reasoning an interviewer expects behind every box and arrow.
1. Clarify requirements before drawing any box
Checkout is a distributed transaction across services that cannot share a single database - inventory, payment, and order fulfillment each own their own data store - which means the core problem is coordinating multiple independent writes so that a failure partway through never leaves the system in an inconsistent state (money taken but no order, or an order with no inventory reserved).
Functional requirements
Non-functional requirements
2. Back-of-the-envelope capacity estimation
These numbers decide how long an inventory hold can safely last, how much saga-state storage is needed for in-flight checkouts, and whether payment calls can be synchronous or must be queued during flash-sale peaks.
| Metric | Assumption | Resulting estimate |
|---|---|---|
| Cart writes | ~50 million active shoppers/day, ~10 cart mutations each | ~500M cart writes/day ≈ 5,800/sec average, ~30,000/sec peak (flash sales) |
| Checkout attempts | ~3% of shoppers complete checkout | ~1.5 million checkouts/day ≈ 17 sagas/sec average, ~800/sec peak |
| Inventory reservation hold duration | 15-minute hold to complete payment before auto-release | At 800 sagas/sec peak, up to ~720,000 concurrent in-flight holds during a sustained flash-sale peak - must be indexed for fast expiry sweeps |
| Payment gateway calls | 1 charge attempt per checkout, some retried on gateway timeout | ~800 calls/sec peak - must respect the external payment processor's own rate limits, so a queue absorbs bursts above that |
| Saga state storage | Each in-flight saga tracks ~5 steps and their status | Modest per-record size; the operational challenge is fast lookup/expiry, not raw storage volume |
3. High-level design (HLD)
The HLD treats checkout as an orchestrated saga: a central orchestrator calls each participating service (inventory, payment, order, shipping) in sequence and is responsible for triggering compensating actions on any downstream failure - no single database transaction spans all of them.
What each box owns
Order orchestrator (saga coordinator)
Drives the checkout as an explicit sequence of steps - reserve inventory, charge payment, create order, hand off to shipping - persisting the outcome of each step to a durable saga log before moving to the next. If any step fails, the orchestrator walks backward through the already-completed steps and invokes each one's compensating action, rather than leaving completed side effects (a charge, a reservation) dangling.
Inventory, payment, and order services
Each is an independent service with its own database, exposing both a "do" action (reserve, charge, create) and a "compensate" action (release, refund, cancel) for the orchestrator to call. None of them know about the other services or the overall saga - they only understand their own local operation and its reversal, which is what keeps them independently deployable and testable.
Saga log & event bus
The saga log is the durable record of exactly which steps have completed for a given checkout - it's what lets the orchestrator (or a recovery process) resume or compensate correctly even after a crash mid-saga, since the log survives independently of any single service's in-memory state. The event bus is how services asynchronously signal step completion/failure back to the orchestrator, decoupling the orchestrator from needing to synchronously block on every downstream call.
Cart service
Deliberately the simplest, most independent service in the system - it just stores line items per buyer and must stay available and fast even if payment or shipping is having a bad day, since abandoning a cart is recoverable but a checkout system that can't even let people browse and add items is not.
4. Detailed architecture diagram
The architecture diagram shows the orchestrator's saga state as durable and replayable, each participant service independently scaled and database-isolated, and the compensation path treated as a first-class, equally-tested code path rather than an afterthought.
| Decision | Choice | Reasoning |
|---|---|---|
| Coordination pattern | Orchestration (central coordinator), not pure choreography | With 4+ services and required compensation ordering, a central orchestrator makes the overall checkout flow and its failure paths explicit and debuggable, versus a web of services reacting to each other's events with no single place showing the whole flow. |
| Saga log durability | Append-only, persisted before each step executes | If the orchestrator crashes mid-saga, a recovery process must be able to read exactly which steps completed and resume or compensate correctly - this is impossible without a durable, step-by-step record. |
| Idempotency | Client-supplied checkout_id, deduplicated at the gateway | A retried checkout request (double-click, client timeout) must resolve to the same saga instance, never start a second one that could double-charge or double-reserve. |
| Compensation failures | Routed to a dead-letter queue for manual/automated ops review | A compensating action can itself fail (e.g. a refund API call times out); silently retrying forever or giving up silently are both unsafe, so unresolved compensations get flagged rather than lost. |
5. Sequence diagrams for the two critical flows
The first diagram shows the happy path from cart to a held inventory reservation; the second shows the full saga including a payment failure and the compensating rollback that follows it.
5.1 Cart checkout & inventory reservation
Step 2's snapshot matters: prices and item availability shown at checkout time are frozen for this saga instance, so a price change elsewhere in the catalog mid-checkout doesn't silently alter what the buyer is charged. Step 3's saga log write happens before step 4's actual call - logging intent before attempting the action is what lets a recovery process later distinguish "we never tried to reserve" from "we tried and don't know if it succeeded," which matters if the orchestrator crashes between steps 4 and 6.
5.2 Order/payment/shipping saga with compensation on failure
Step 3's decline is a completely ordinary, expected outcome - not a system failure - which is exactly why the saga treats it as a normal branch with a defined compensation path rather than an exception to handle ad hoc. Step 5's compensation only has to undo what actually succeeded (the inventory reservation); because the saga never reached the order or shipping steps, there's nothing to compensate there - this is why the saga log's step-by-step record in step 4 is what tells the orchestrator exactly how far back it needs to unwind, rather than blindly compensating every possible step.
6. Entity-relationship (ER) diagram and schema
The schema has to answer: how does an order remain traceable to the exact cart snapshot it was created from, how is inventory held without a hard cross-service transaction, and how is a payment attempt linked to exactly one order without ambiguity.
Key modeling decisions
| Storage choice | Use when | Watch out for |
|---|---|---|
| Relational (Postgres/MySQL) for orders and payments | Strong consistency and unique constraints (checkout_id, idempotency) are required for financial correctness. | saga_state as JSON is convenient but should stay a small, bounded structure - not a place to store growing logs. |
| Key-value / cache (Redis) for cart line items | Access pattern is simple per-buyer reads/writes at very high frequency, tolerant of the cart being rebuilt if lost. | Carts should periodically snapshot to a durable store too, so a logged-in buyer's cart survives a cache eviction across sessions. |
7. Deep dives interviewers actually probe
Saga (orchestration) vs a single distributed transaction (2PC) - why not just use one database transaction?
Two-phase commit would require every participant (inventory, payment, order) to hold locks open across a network round-trip to a coordinator, across services that in practice are often owned by different teams and sometimes different companies entirely (the payment gateway is external and simply doesn't support being a 2PC participant). A saga trades strict atomicity for eventual consistency with explicit, tested compensating actions - each step commits locally and independently, and failures are corrected afterward rather than prevented by holding a lock across the whole flow.
What happens if the compensating action itself fails (e.g. the refund call times out)?
Compensating actions are retried with exponential backoff, since most failures (network blip, transient gateway error) resolve on retry. If retries are exhausted, the saga is marked into a distinct compensation_failed state and routed to a dead-letter queue for operator or automated reconciliation review - it is never silently marked "failed" and forgotten, since that could mean a buyer was charged with no order to show for it.
How does the inventory hold avoid becoming a source of oversold or perpetually-locked stock?
Every reservation carries an explicit expiry (e.g. 15 minutes); a background sweeper periodically finds reservations past their expiry with no corresponding completed order and releases the held quantity back to available_qty. This bounds the worst case - a buyer who reserves items and then abandons checkout entirely (no compensation ever triggered because the saga never explicitly failed, it just went silent) still can't lock inventory forever.
Why is the shipping step never reached in the failure example - couldn't it start in parallel to save time?
The steps have a hard dependency order for a business reason, not just a technical one: shipping physical inventory before payment is confirmed captured risks shipping goods that are never paid for, which is a real financial loss, not just a data-consistency inconvenience. Some sagas do parallelize independent steps (e.g. sending a confirmation email alongside creating the shipping label) but payment-before-fulfillment is treated as a strict sequential gate.
What is the single biggest bottleneck as this scales 10x?
Not the orchestrator itself - it's largely stateless per-saga-step logic and scales horizontally. The real bottleneck becomes the external payment gateway's own rate limits during a flash sale, since that's a third party outside this system's control. The fix is a request queue in front of the payment call that smooths bursts to match the gateway's sustained rate, combined with surfacing "processing your payment" state to the buyer rather than either blocking synchronously or failing outright when the gateway is temporarily saturated.
Post a Comment
Add