Amazon Great Indian Festival: Checkout Infrastructure That Handles a 50x Traffic Spike

Amazon Great Indian Festival: Checkout Infrastructure That Handles a 50x Traffic Spike

1. Clarifying Requirements

1.1 Functional Requirements

• Customers add items to a cart, start checkout, choose address and delivery option, apply offers (coupons, bank offers, Prime benefits, gift cards, loyalty), pay, and receive an order confirmation. Every step must work during the festival, and the last steps (payment and order placement) must work best.

• Checkout serves tens of millions of items across millions of sellers, with deal prices that go live at specific moments (midnight launches, Prime early access, hourly lightning deals, category "price drops").

• The system computes price, tax, shipping fee, discounts, and EMI/no-cost-EMI breakups consistently, and locks the price the customer saw for a bounded time.

• The system reserves inventory at the right moment so that limited-quantity deals aren't oversold, while ordinary items sell without needless friction.

• Checkout supports UPI, credit and debit cards (with 3-D Secure), net banking, wallets, EMI, gift cards, pay-on-delivery (COD), and instant bank discounts that depend on the card, issuer, and order value.

• Payments can be slow, ambiguous, duplicated, or late. The system must not double-charge, lose paid orders, or leave money with no order.

• The system creates an order exactly once per customer intent, even when the customer double-taps, the app retries, or the gateway sends duplicate callbacks.

• After order placement, the system triggers fulfilment, seller notification, invoicing, loyalty updates, and customer notifications. These downstream steps may lag during peak, but must not be lost.

• Customers can review, modify (address, payment method within limits), and cancel orders for a window after placement, and see accurate order status.

• The system supports fraud and risk screening (account takeover, card testing, COD abuse, promo abuse, reseller behaviour) without blocking genuine customers.

• Operators can pre-scale, shift traffic, disable features by kill switch, throttle noncritical work, change payment routing, and monitor the checkout funnel in real time.

1.2 Non-Functional Requirements

• The spike is 50x baseline and arrives in steps, not as a ramp. Midnight on day one, hourly deal drops, and noon "blockbuster" windows produce near-vertical increases. Autoscaling that reacts in minutes is too slow, so the platform must be pre-scaled to the forecast peak plus headroom and designed so that the unscaled parts degrade gracefully.

• Checkout is the revenue path, so it gets the strongest protection. Browse, search, recommendations, reviews, and ads may degrade. Cart, price lock, payment, and order creation must stay up. This implies explicit dependency tiering and priority-based load shedding.

• Correctness requirements:

  • No double orders from retries, and no double charges.

  • No lost paid orders: every successful payment ends in an order or a refund.

  • No material oversell on limited deals, and bounded, well-handled oversell elsewhere.

  • Price integrity: the customer pays what checkout displayed, within the lock window, and promos can't exceed their budgets.

• Latency targets under peak: cart operations p99 < 300 ms, checkout page assembly p99 < 800 ms, payment initiation p99 < 1 s (excluding the customer's bank steps), order creation p99 < 1 s. Tail latency matters more than the median, because a checkout makes many sequential calls.

• Availability: 99.99% for the checkout path during the event, with graceful partial failure (for example, COD disabled or a payment method hidden) instead of a total outage.

• Retry amplification is the biggest hidden risk. A 20% slowdown at one dependency causes client and service retries that can multiply load several-fold, so retries need budgets, jitter, and idempotency.

• Blast-radius containment. A bad deploy, a hot SKU, a failing payment gateway, or a slow seller service must not take down checkout for everyone. Cells, bulkheads, and shuffle sharding limit the damage.

• Freshness versus stability. Deal prices and stock must be fresh enough to be correct, but not so fresh that every request hits the source of truth. The design chooses bounded staleness with authoritative checks at the reservation and payment steps.

• Cost efficiency. Capacity is provisioned for a few days a year, so the design reuses elasticity where it is real (stateless tiers, caches), avoids over-provisioning stateful tiers by sharding well, and uses cheap, async paths for everything non-critical.

• Observability and operability. Real-time funnel metrics, per-dependency saturation, synthetic checkout probes, a war-room process, change freezes, and pre-written runbooks, because the event is short and mistakes are expensive.

• Compliance and security. PCI-DSS scope minimisation (tokenised card data), RBI rules for payments (tokenisation, 3-D Secure, UPI limits, COD policies), tax and invoicing rules, and data protection.

2. Back-of-the-Envelope Estimation

Baseline and peak traffic:

• Assume a normal-day peak of ~150K site requests/sec and ~250 orders/sec. Festival peak at 50x is ~7.5M site requests/sec, mostly browse, search, and product pages (served largely from CDN and caches).

• Orders do not scale 1:1 with traffic. Assume ~12,000 orders/sec at the midnight peak (about 48x baseline), with bursts to ~18,000/sec in the first seconds of a marquee deal.

• Funnel ratios at peak: ~5 cart events per order, ~4 checkout starts per order (many abandon at payment), ~1.35 payment attempts per order (retries and method switches). That gives ~60K cart operations/sec, ~48K checkout starts/sec, and ~16K payment attempts/sec.

Request amplification inside checkout (the number that matters):

• A checkout start fans out to: cart, pricing and promotions, tax, address and serviceability, delivery promise, inventory check, offers eligibility, fraud score, payment-method availability, loyalty and wallet balance. Call it 12-15 internal calls per checkout start, of which maybe 6-8 are on the critical path.

• 48K checkout starts/sec × ~14 calls ≈ ~670K internal calls/sec from checkout alone, before retries. With a 30% retry rate during degradation, that rises to ~870K calls/sec.

• If each call has a p99 of 100 ms, a sequential chain of 7 critical calls has a p99 on the order of 700 ms, and a single slow dependency pushes the whole checkout past its budget. This is why the critical path must be short, parallel, and tolerant of dependency failure.

Hot SKU contention:

• A marquee phone with 40,000 units on a midnight deal can attract 150K-250K reservation attempts/sec in the first ten seconds. One in-memory counter handles that rate, but one database row would handle only ~700 updates/sec, so the hot-SKU path must not use per-order row updates in a shared database.

• Across the deal catalogue, ~5,000 SKUs are "hot" (above 1K attempts/sec at some point), and the other ~tens of millions are cold. The design must be dual-mode.

Inventory and reservation volume:

• 12K orders/sec × ~2.2 items ≈ 26K reservations/sec on average across SKUs. Hot SKUs concentrate, so the in-memory owner tier needs to handle ~300K operations/sec in aggregate with microsecond operations, while durable writes are batched.

• Reservation TTL of ~10 minutes (long enough for 3-D Secure and UPI approval), with concurrent live reservations: 12K orders/sec × ~35% unconverted × 600 s ≈ ~2.5M live reservations. Small in memory (~100 bytes each is ~250 MB), so it is a sizing non-issue.

Pricing and promotions:

• Price computation per cart view (~5 items) at 150K cart views/sec = ~750K price lookups/sec, which must be served from precomputed, cached deal prices, with promo evaluation in memory.

• Bank-offer budgets: a "10% instant discount on Bank X cards, up to ₹1,500, budget ₹50 crore" offer is a global counter. At peak, ~3,000 redemptions/sec on one budget counter requires a sharded counter design (section 6.3).

Payments:

• ~16K payment attempts/sec across UPI, cards, net banking, wallets, and EMI. A single gateway route typically sustains ~1-3K TPS, so the platform needs 10-20 concurrent routes across PSPs and banks, health-weighted.

• Failure rates at peak: 8-15% of attempts time out or fail at the bank, and 1-3% end in an ambiguous state. That is ~160-500 ambiguous payments/sec to be resolved by polling, callbacks, and reconciliation.

• COD share ~25-30% of orders (~3,500 orders/sec), with a risk check per order.

Order store:

• 12K orders/sec × ~6 KB (items, address snapshot, price breakup, payment refs) ≈ 72 MB/sec of order writes (~260 GB/hour at peak), plus order-event fan-out of 10-15 events per order (150K events/sec) to downstream consumers.

• Sharded by customer across ~256-512 logical shards, so each logical shard takes ~25-50 orders/sec at peak, which is comfortable.

Downstream fan-out:

• Fulfilment, seller, invoice, loyalty, notification, analytics: ~150K events/sec on the stream, with consumers allowed to lag by minutes. Notification volume: 12K orders/sec × 2-3 messages = ~30K messages/sec (push is cheap, SMS and email need throttling and priority).

Cache sizing:

• Deal prices and promo rules for ~5M active offers at ~500 bytes: ~2.5 GB, replicated per cell and region. Trivial in size, critical in hit ratio (target above 99.5%).

Capacity planning rule of thumb:

• Provision stateless tiers at ~2x the forecast peak (to absorb forecast error and one cell loss), stateful tiers at ~1.5x with verified headroom, and verify with a load test at 1.3-1.5x forecast before the event.

Cost of failure:

• Each minute of checkout outage at 12K orders/sec at an average order value of ₹1,800 is ~₹130 crore of order value per minute at risk, so the engineering budget for resilience is easily justified.

3. Core APIs

Cart

POST /v1/cart/items
Headers: { Authorization, Idempotency-Key }
Body: { item_id, seller_offer_id, qty: 1 }
Response: { cart_id, version: 17, items: [ { offer_id, qty, display_price, deal_ends_at } ], subtotal_hint, price_staleness_ms: 800 }

GET /v1/cart            → cart with cached display prices (non-authoritative until checkout)
PATCH /v1/cart/items/{offer_id}   Body: { qty }

Start checkout (authoritative pricing, price lock)

POST /v1/checkout/sessions
Headers: { Authorization, Idempotency-Key }
Body: { cart_id, cart_version: 17, address_id, delivery_pref: "STANDARD", offers: { coupon: "FEST10" } }
Response: {
  checkout_id: "co_8f2…",
  lines: [ { offer_id, qty, unit_price, tax, discount_breakup, delivery_promise: "2025-10-04" } ],
  totals: { items: 41980.00, shipping: 0.00, discounts: -3000.00, tax_incl: 6403.39, payable: 38980.00 },
  price_lock: { token: "pl_…", expires_at: "…+15m" },
  payment_options: [ { method: "UPI", available: true }, { method: "CARD", offers: [ { bank: "XBANK", instant_discount: 1500, remaining_budget_band: "HIGH" } ] }, { method: "COD", available: false, reason: "RISK" } ],
  degraded: { offers_unavailable: false }
}

Reserve inventory and place order (single idempotent call)

POST /v1/checkout/{checkout_id}/place
Headers: { Idempotency-Key: "{customer}:{checkout_id}" }
Body: { payment: { method: "CARD_TOKEN", token_ref, instrument_id }, price_lock_token, device_signals }
Response: {
  order_intent_id: "oi_…",
  status: "PAYMENT_REQUIRED" | "ORDER_CONFIRMED" | "PENDING_CONFIRMATION" | "REJECTED",
  payment: { payment_id, next_action: "REDIRECT_3DS" | "UPI_APPROVE" | "NONE", expires_at },
  reservation: { held_until: "…+10m", lines_reserved: 2, lines_unavailable: [] },
  order_id?: "404-1234567-8901234"
}

Payment result and status (customer-facing, resilient to lost callbacks)

GET /v1/orders/intent/{order_intent_id}
Response: { state: "RESERVED"|"PAYMENT_PENDING"|"CONFIRMED"|"PENDING_CONFIRMATION"|"FAILED_REFUND_INITIATED"|"EXPIRED",
            order_id?, message: "We’re confirming your payment. You will not be charged twice.", poll_after_ms: 2000 }

Order APIs

GET  /v1/orders?cursor=…              → paged, from the order read model (eventually consistent, ≤ few seconds)
GET  /v1/orders/{order_id}
POST /v1/orders/{order_id}/cancel     Body: { reason }       → idempotent, state-machine guarded

Payment provider callbacks (internal, idempotent)

POST /internal/v1/payments/callbacks/{psp}
Body: { psp_txn_id, order_intent_id, status, amount, bank_ref, event_time }
Response: { received: true }                       // persisted to an inbox, processed asynchronously

Operations and control

PUT  /internal/v1/traffic/priorities     Body: { classes: { CHECKOUT_PLACE: 1, PAYMENT: 1, CART: 2, PDP: 3, SEARCH: 4, RECS: 5 } }
POST /internal/v1/features/{flag}        Body: { enabled: false, scope: "CELL:7" }       // kill switches per cell
PUT  /internal/v1/payments/routing       Body: { method: "CARD", weights: { psp_a: 40, psp_b: 35, psp_c: 25 }, issuer_overrides: {...} }
PUT  /internal/v1/inventory/hot-skus     Body: { sku: "SKU123", mode: "HOT_OWNER", shards: 64 }
GET  /internal/v1/checkout/funnel/live   → stage counts, conversion, p99 per stage, dependency saturation, ambiguous-payment backlog

4. High-Level Design

4.1 Design Principles

Components needed:

• Protect the checkout path with tiers. Tier 0 (cart, price lock, inventory reservation, payment, order creation) gets reserved capacity, strict timeouts, no optional dependencies, and the best engineers' attention. Tier 1 (delivery promise, offers, address validation) has degraded fallbacks. Tier 2 (recommendations, ads, reviews, loyalty widgets) is shed first.

• Shorten and parallelise the critical path. Every synchronous dependency in the checkout sequence multiplies failure probability and tail latency. The design does as much as possible before checkout (cached, precomputed data) and after order placement (async events).

• Accept the order durably, finish the work asynchronously. Once payment is authorised and the reservation is held, the order is written to a durable log and acknowledged. Invoicing, seller notification, fulfilment planning, and loyalty happen from events, so downstream slowness doesn't block revenue.

• Pre-scale and pre-warm, don't autoscale into the spike. Capacity, caches, connection pools, and JIT-sensitive runtimes are warmed before the event, and autoscaling is a backstop for forecast error.

• Cells, not one big fleet. The platform is partitioned into independent cells (each a full stack serving a slice of customers), so that a failure or a bad deploy affects 1/N of customers, and capacity is added by adding cells.

• Idempotency everywhere. Every state-changing call has an idempotency key, and every consumer is safe to replay, because retries and redeliveries are certain at this scale.

• Degrade by priority, never by randomness. Load shedding follows a pre-agreed order, with explicit user-facing behaviour at each step, and the shedding is automated.

The architecture has: edge and traffic management, cart service, pricing and promotion services, checkout orchestrator, inventory reservation, payment orchestration, order service, asynchronous downstream pipeline, risk and fraud, and resilience, capacity, and operations tooling.

4.2 Edge, Traffic Classes, and Admission Control

Components needed:

CDN and edge absorb browse traffic. Product pages, images, search suggestions, and static assets are served from the CDN and edge caches with long TTLs and stale-while-revalidate, so that 7.5M requests/sec of site traffic reaches the origin at a small fraction. Deal prices on product pages are cached for a few seconds, with the authoritative price confirmed at checkout.

Traffic classification and priority. Every request is tagged with a class at the edge (CHECKOUT_PLACE, PAYMENT, CART, PDP, SEARCH, RECS, ADS, etc.). Under load, the edge and the service mesh enforce priority-aware admission: lower classes are throttled or served stale first, and the top classes keep reserved capacity pools in each service. This is implemented by per-class concurrency limits (adaptive, based on measured latency), not by global random drops.

Per-customer rate limits and bot controls. Per-account, per-device, and per-IP limits (tolerant of shared mobile NATs), bot scoring, and challenge steps for suspicious sessions protect cart and checkout from scripted traffic (deal sniping, cart-hoarding, coupon farming). Checkout endpoints have stricter limits than browse.

Targeted waiting rooms, not a global one. For a handful of ultra-hot deals (a flagship phone with a few thousand units), a deal-level admission gate limits how many customers enter checkout for that SKU concurrently, using the same stateless or queued admission patterns as a flash sale. This protects the inventory owner and the payment tier from a stampede on one SKU, while all other checkout traffic flows normally. It is a surgical tool, not the default.

Staggered deal starts. A business-level load-shaping lever: deals across categories go live at slightly different times (for example staggered by 30-90 seconds, or in rolling waves by category and region) instead of all at 00:00:00, which turns one vertical spike into several smaller steps. Customers see a countdown per deal, and the deal calendar is planned with capacity in mind.

Client behaviour. The app and web client back off with jitter, obey Retry-After, avoid auto-retry storms on 5xx, and use a single in-flight "place order" request per checkout (the button disables, and the idempotency key is reused on retries).

Regional routing and cell assignment. Customers are mapped to a home cell (by hashed customer ID, sticky), so their cart, checkout sessions, and recent orders are co-located. The edge routes to the home cell, and if the home cell is unhealthy, traffic fails over to a designated neighbour cell with the necessary data replicated.

4.3 Cart Service: Cheap, Scalable, and Not Authoritative on Price

Components needed:

Cart storage. A key-value store keyed by customer (and a guest-cart token for logged-out users), holding line items, quantities, offer IDs, and a version. Writes are single-key, so the store scales horizontally with no hot partitions (a customer's cart is rarely written at more than a few operations per second). Carts live in the customer's home cell, with asynchronous cross-cell replication for failover.

Cart reads are cached and priced from precomputed data. A cart view needs display prices for ~5 items. Prices come from the deal price cache (precomputed and pushed ahead of the event), not from live promotion evaluation. The cart shows "price as of a few seconds ago", with the authoritative price computed at checkout start.

Add-to-cart under spike conditions. At ~60K operations/sec, add-to-cart must stay cheap: no inventory call, no promo call, no tax call. It validates that the offer exists and is active (from a cached catalogue snapshot), writes the cart, and returns. A "this deal may sell out" hint can be derived from cached stock bands, but there's no reservation at add-to-cart (reserving at cart time would let hoarders and abandoners lock stock).

Cart merge and cleanup. Guest-to-user merge, item-level deduplication, and removal of expired deals happen lazily on read, with background cleanup, so spikes don't generate cleanup storms.

Degradation behaviour. If the cart store is degraded, the app can fall back to a local cart (client-side) with reconciliation at checkout, so a cart outage doesn't block purchasing intent.

4.4 Pricing, Promotions, and Price Lock

Components needed:

Precomputed deal prices. Deal definitions (item, seller offer, price, start and end times, per-customer limits, eligibility such as Prime-only) are authored days ahead, validated, and published to a versioned deal snapshot that is pushed to every cell's cache before the event. At go-live time, the price flips by time-based activation (each node compares the server time against starts_at), so there's no thundering write at 00:00:00 and no central switch that can fail.

Promotion engine. Combines item deals, coupons, bank offers, Prime benefits, loyalty and gift-card credits, and shipping rules into a final price, under stacking rules (which discounts combine). Rules are compiled into an in-memory form on each node, so evaluating a cart takes microseconds. The engine returns the line-level breakup, which is shown to the customer and stored on the order.

Tax and shipping. Tax rules (GST by HSN code, origin and destination states, seller registration) and shipping fees (by weight, distance, Prime status, order value thresholds) are computed from cached rule tables, with the rule version recorded on the checkout session.

Price lock at checkout start. POST /checkout/sessions computes the authoritative totals and issues a signed price-lock token valid for ~15 minutes, containing the line prices, discounts, tax, and the rule versions. At placement, the system verifies the token instead of recomputing from live sources. This means a customer who started checkout at ₹38,980 isn't surprised by a ₹39,500 price when the deal ends mid-checkout (within the lock window and subject to inventory), and the placement path needs no second round of pricing calls.

Offer budgets and per-customer limits. Bank offers and coupons have global budgets and per-customer limits. Reserving budget at checkout start for hot offers is impractical (it would lock budget for abandoners), so the budget is consumed at order placement by a sharded counter (section 6.3), and the checkout page shows only a coarse remaining-budget band (HIGH, LOW, EXHAUSTED). If the budget runs out between page view and placement, the order proceeds at the undiscounted price only with explicit customer confirmation (never silently), or the offer is honoured under a documented overshoot tolerance for small overshoots.

Fallbacks. If the promotion engine is unavailable, checkout falls back to the price lock from the cart's cached price with offers disabled, and shows "some offers are temporarily unavailable". It does not block the purchase.

4.5 Inventory Reservation: Dual-Mode for Cold and Hot SKUs

Components needed:

What "inventory" means. Available-to-promise (ATP) is computed per SKU per fulfilment node (warehouse or seller location), considering on-hand stock, in-transit, reservations, and seller-confirmed capacity. At peak, the checkout path needs a fast, reservable ATP count, not the full supply-chain truth.

When to reserve. The design reserves at order placement (the place call), not at cart or checkout start. Reserving earlier would lock stock for abandoners (at ~65% abandonment, an early reservation model would make stock look 3x more depleted than reality), and reserving later (after payment) risks charging customers for items that are gone. Reservation at placement, with a TTL of ~10 minutes while payment completes, is the compromise. For limited-quantity deals, the reservation is strict and counted against the deal's quota. For ordinary items, the reservation is soft (a limited, tolerated oversell, handled by backorder or cancellation).

Cold SKUs (the vast majority). Stored in a horizontally partitioned key-value store keyed by (sku, node), using conditional decrements (available ≥ qty), which scale because contention per key is low. A reservation record (reservation_id, sku, node, qty, expires_at) is written transactionally with the decrement.

Hot SKUs (a few thousand at the event). Per-key database conditional writes can't sustain 150K-250K attempts/sec. For SKUs flagged hot (by pre-event forecast and by live detection), reservation moves to in-memory single-writer owners, sharded by SKU (and for the very hottest, by (SKU, shard) with the quota split into sub-quotas):

• The owner holds the counter in memory, processes requests in micro-batches, appends decisions to a replicated log (group commit), and acknowledges.

• Overshoot protection: a sub-quota shard that runs dry borrows from a sibling through a small rebalancing protocol, so the SKU can sell out fully without stranding units in an idle shard, while preserving sum(sold + held) ≤ quota.

• The durable store is updated asynchronously from the log, and a unique reservation ID per (checkout, line) makes replay safe.

Hold lifecycle. RESERVED → CONFIRMED on payment success (the order is created), or EXPIRED/RELEASED on failure, abandonment, or timeout. A sweeper plus TTL expiry guarantees release, and a release event feeds a short-lived "back in stock" signal for waiting customers.

Overselling policy. For hot limited deals, strict (the owner is authoritative). For regular items, the system can allow a small, configured oversell ratio (for example 1-2% for items with fast replenishment) because the cost of lost sales exceeds the cost of occasional cancellations, and the policy is per category and seller-tier.

Failure behaviour. If the reservation service is slow or down, checkout fails closed for limited deals (no order without a reservation) and fails open with soft inventory for ordinary items (accept the order, then confirm stock asynchronously, cancelling and refunding if the seller can't fulfil), under a global kill switch with a cap on the open-failure volume.

Multi-line orders. A checkout with multiple lines reserves each line independently, in a deterministic order, with compensation (release) if any strict line fails, and the customer is shown which lines are unavailable with options to proceed with the rest.

4.6 Checkout Orchestrator: A Short, Resumable State Machine

Components needed:

State machine. SESSION_CREATED → PRICE_LOCKED → PLACED → RESERVED → PAYMENT_PENDING → PAID → ORDER_CREATED, with failure branches (RESERVATION_FAILED, PAYMENT_FAILED, EXPIRED) and a PENDING_CONFIRMATION state for ambiguity. The orchestrator persists each transition to a durable store (keyed by checkout_id), so any node can resume it after a crash.

The synchronous path at placement (target p99 < 1 s excluding the customer's own bank steps):

1. Validate the idempotency key and the price-lock token (local signature check).

2. In parallel: reserve inventory, run a fast risk check (precomputed customer risk plus a lightweight real-time model with a tight timeout and a safe default), and verify the payment instrument (token valid, method available).

3. Create the payment intent with the chosen route.

4. Return the next action to the client (redirect for 3-D Secure, UPI app approval, or immediate confirmation for COD or wallet).
   Everything not essential to those steps (delivery promise recalculation, loyalty, invoice, recommendations) is excluded.

Saga with compensation. If payment fails or the reservation fails, compensation actions (release reservation, void authorisation, release offer budget) run idempotently, driven by the state machine and a retry queue, not by in-memory try/catch, so a crash mid-compensation doesn't leak stock or budget.

Deadlines and timeouts propagated downward. Each call carries a deadline (the remaining time budget). Services that can't finish within it fail fast instead of consuming capacity on work the caller has abandoned, which is crucial for stopping retry storms from compounding.

Concurrency limits and bulkheads. The orchestrator uses per-dependency concurrency limits (semaphores), separate thread and connection pools, and circuit breakers, so a slow risk service can't consume all orchestrator threads. Every dependency has an explicit fallback behaviour (for example risk service timeout → treat as medium risk and allow with extra verification for high-value orders; promo engine timeout → no offers; delivery promise timeout → show a default estimate).

Hedged and parallel calls. For idempotent reads (risk precomputation, delivery estimates), the orchestrator may issue hedged requests to a second replica after a p95 delay, to cut tail latency, bounded by a retry budget.

4.7 Payment Orchestration

Components needed:

Why payments dominate failure risk. At ~16K payment attempts/sec, the platform depends on dozens of external systems (PSPs, acquirer banks, issuer banks, UPI rails, 3-D Secure access control servers), each with its own capacity limits and failure modes, and the platform controls none of them.

Routing layer. A payment router chooses a route (PSP and acquirer path) per attempt using: method, issuer bank, card network, amount, and real-time route health (success rate, latency, timeouts, error-code mix) over sliding windows with minimum sample sizes. Weights adapt gradually, with per-route concurrency caps, circuit breakers, and a small exploration share so alternatives stay warm. Issuer-level overrides handle bank outages: if Bank X's cards fail on route A, traffic for Bank X shifts to route B, and the customer sees an accurate message if no route works ("Bank X is currently facing issues, try another method").

Payment methods:

• UPI: collect and intent flows, with polling and callbacks. UPI is the highest-volume method and has its own rail limits, so UPI routing spans multiple PSP and bank handles.

• Cards: tokenised credentials (network tokens), 3-D Secure with frictionless flows where issuers allow, and instant bank discounts applied at authorisation (discount funded by the bank and platform per the offer terms).

• Net banking and wallets: redirect flows with per-bank health tracking.

• EMI and no-cost EMI: the EMI plan and subvention are part of the price lock, with bank-side eligibility calls kept off the placement critical path where possible (eligibility is precomputed at checkout start, cached briefly).

• COD: no payment step, but a risk gate (customer history, address risk, order value limits, pin-code policy) and optional confirmation (OTP or a call) for high-risk orders.

Payment state machine and ambiguity. INITIATED → AWAITING_CUSTOMER → AUTHORIZING → AUTHORIZED | DECLINED | UNKNOWN. A timeout is UNKNOWN, not failure. Callbacks land in an inbox deduplicated by (psp, txn_id, status), and are processed asynchronously. The orchestrator also polls status with backoff, and reconciles against provider settlement files. The customer sees "confirming your payment, you won't be charged twice", and the order is held in PENDING_CONFIRMATION until a definitive answer arrives.

Hold interplay. The inventory reservation TTL (~10 minutes) covers normal payment latency. If payment is confirmed to be in progress at TTL expiry, the reservation is extended once, briefly, for non-hot SKUs. For hot limited deals, the extension is short or none, with a late-success path: re-reserve if units remain, otherwise refund automatically.

Late and duplicate confirmations. A success callback after the reservation was released tries to re-reserve (if stock remains), otherwise auto-refund. A second success for the same intent (customer paid twice via two methods) triggers an automatic refund of the surplus. Every outcome is final within a bounded time and visible to the customer.

Money ledger. A double-entry ledger records authorisations, captures, voids, refunds, discounts funded by banks, and platform subsidies, and reconciles daily against PSP, bank, and settlement records.

4.8 Order Service: Idempotent, Sharded, and Event-Driven

Components needed:

Order creation. When payment is AUTHORIZED (or COD passes its risk gate), the orchestrator creates the order with a unique constraint on (customer_id, checkout_id), so retries, duplicate callbacks, and replays produce exactly one order. The order record includes an immutable snapshot of items, address, price breakup, tax details, offers applied, payment references, and the reservation IDs, so later changes to catalogue or promotions can't alter history.

Sharding. Orders are sharded by customer ID (so a customer's orders are co-located and "my orders" is a single-shard read), with shard-prefixed order IDs for routing without a lookup. Writes at ~12K orders/sec spread over hundreds of logical shards.

Transactional outbox. The order write and its events (OrderPlaced, InventoryConfirmed, PaymentCaptured) are committed in the same transaction into an outbox table, which a relay publishes to the event stream. This gives at-least-once publication with no dual-write gap, and consumers are idempotent.

Order state machine. PLACED → CONFIRMED → PACKED → SHIPPED → DELIVERED, with CANCELLED, RETURN_*, and REFUNDED branches, guarded by compare-and-set on a state version. Cancellation during the festival's high-volume window is a common request and must be cheap and safe (release stock, void or refund, notify sellers).

Order read model. "My orders" and order detail pages are served from a denormalised, eventually consistent read model (a few seconds behind), cached per customer, so order history reads never touch the write path. Immediately after placement, the confirmation page reads from the write path or from a short-lived per-customer cache to give read-your-writes.

Order confirmation under pressure. If downstream systems are slow, the customer still sees "Order placed" because the order record and payment confirmation are durable. Invoice generation, seller notification, and delivery slot assignment follow asynchronously. This is the key decoupling that lets the system accept orders at a rate higher than downstream systems can process.

4.9 Asynchronous Downstream Pipeline and Backpressure

Components needed:

Event stream and consumers. Order events flow into a partitioned log (keyed by order or customer). Consumers include fulfilment planning, seller and warehouse notification, invoicing and tax documents, loyalty and rewards, customer notifications, fraud post-processing, analytics, and finance. Each consumer has its own consumer group, lag monitoring, and retry and dead-letter handling.

Allowed lag, by consumer. Fulfilment planning: minutes. Invoicing: tens of minutes. Notifications: seconds to minutes (with priority). Analytics: hours. Consumers can fall behind during the spike and catch up afterward, which is the point of the log. Lag alarms trigger autoscaling of consumers, and priority lanes keep critical consumers (fulfilment) ahead of low-priority ones.

Backpressure and shedding. If a downstream system is overwhelmed (a seller notification gateway, the invoicing service), the stream buffers. Retention is sized to hold many hours of peak traffic, and consumer-side rate limits protect the downstream. Non-critical consumers can be paused entirely (analytics enrichment, recommendation updates) to free capacity.

Notifications. Push is the primary channel (cheap, fast). SMS and email are throttled to provider capacity, prioritised (payment confirmation and delivery OTPs first, marketing never), deduplicated by event, and rate-limited per customer. The order confirmation screen is the source of truth, so a delayed SMS doesn't leave customers uncertain.

Idempotent consumers and replays. Every consumer keys its effects by (order_id, event_type) so replays don't double-invoice or double-notify, and the platform can replay a time range after a fix.

4.10 Risk and Fraud: Fast, Layered, and Fail-Safe

Components needed:

Precompute what you can. Customer risk scores, device reputation, address risk, and payment-instrument reputation are computed continuously and cached, so the checkout-time check is a lookup plus a lightweight model evaluation (a few milliseconds).

Layered decisions at placement. Allow, step-up (extra authentication, OTP for COD, 3-D Secure forced), hold for review (order accepted, fulfilment delayed pending checks), or decline. The risk call has a tight timeout (for example 80 ms) with a safe default (allow with a standard verification level for low-value orders, step-up for high-value), so a slow risk service never blocks placement.

Festival-specific abuse patterns: coupon and bank-offer farming (many accounts, one card), reseller bulk buying of deal items, card testing through small orders, COD refusal abuse, and account takeover. Per-customer purchase limits on deal items, instrument and address velocity limits, and graph-based post-order review catch most of these without adding latency at placement.

Post-order controls. Orders flagged after placement (by async analysis) are held before shipment, cancelled with a refund if confirmed abusive, and the freed deal stock is returned to the pool. The sync path is permissive within limits, and the async path is thorough.

4.11 Cells, Capacity, Resilience, and Degradation

Components needed:

Cell-based architecture. The checkout stack is deployed as many independent cells, each containing the cart, checkout orchestrator, pricing cache, reservation proxies, payment orchestrator, and order shards for a slice of customers. Customers map to a cell by hashed ID. Benefits:

• Blast radius: a bad deploy, a poison request, or a saturated dependency affects one cell, so ~1/N of customers.

• Capacity scaling by replication: add cells instead of scaling a monolith, and each cell has a known, load-tested capacity (for example 1,200 orders/sec).

• Failure isolation: a cell failure fails over to a neighbour with replicated state, within its headroom.

• Safe deployment: changes roll out cell by cell with bake time, never everywhere at once (and a change freeze applies during the event).
  Shuffle sharding for shared dependencies (for example the risk service, the notification gateway): each cell or customer group uses a different random subset of dependency instances, so one poisoned client or one failing instance affects only the few that share that exact subset.

Pre-scaling and warm-up.

• Capacity for stateless tiers is provisioned to ~2x forecast peak before the event, in each cell and region, and stateful tiers (stores, owners, streams) are sized to ~1.5x with verified headroom.

• Caches are primed: deal snapshots, promo rules, tax tables, catalogue slices for deal items, and payment routing health.

• JVM and runtime warm-up, connection-pool pre-establishment, TLS session caches, and DNS pre-resolution are done before traffic arrives, since cold nodes under a vertical spike are a classic failure mode.

• Autoscaling is configured as a backstop with fast scale-out and conservative scale-in (don't scale in until the event is over).

Load testing and game days. Full-stack tests at 1.3-1.5x forecast peak run in production-like environments (and, where safe, in production using synthetic traffic with test tenants), including: hot-SKU contention, payment route failure, a slow dependency (latency injection), cache flush, a cell loss mid-peak, and a retry storm. Findings are fixed or mitigated with kill switches before the event.

Dependency tiering and static fallbacks. Every dependency of checkout is classified:

• Must-have (inventory reservation for limited deals, payment, order store): redundancy, reserved capacity, tight SLOs.

• Should-have (offers, delivery promise, tax detail): bounded timeouts with cached or default fallbacks.

• Nice-to-have (recommendations, loyalty projections, ads): removed from the critical path or shed first.

Degradation ladder (ordered, automated where possible):

1. Serve stale prices and stock bands on browse and cart pages (longer TTLs).

2. Shed Tier 2 features (recommendations, "customers also bought", ads) from checkout and cart pages.

3. Disable optional promotion evaluation (coupons and bank offers fall back to the locked price, with "offers temporarily unavailable" messaging).

4. Replace live delivery promise with a cached estimate by pincode.

5. Disable non-critical payment methods or routes with poor health (EMI eligibility checks, certain wallets), with clear messaging.

6. Enable deal-level admission gates for the hottest SKUs.

7. Switch ordinary-SKU reservation to soft/async confirmation (bounded oversell).

8. Pause low-priority async consumers to protect shared infrastructure.

9. Last resort: serve a static "high demand, try again shortly" experience for part of the traffic in affected cells, with fast recovery, instead of cascading failure.

Retry and overload control. Retry budgets (retries limited to a percentage of base traffic), exponential backoff with jitter, deadline propagation, adaptive concurrency limits at each service (reject quickly when saturated, rather than queueing indefinitely), and circuit breakers with half-open probes.

4.12 Observability, Operations, and the War Room

Components needed:

• Real-time funnel: cart adds, checkout starts, placements, payments initiated, authorised, orders created, with conversion and latency per stage, by cell, region, payment method, and issuer. Anomalies in a stage's conversion are the first signal of a hidden failure.

• Dependency saturation view: concurrency, queue depth, error rates, and p99 per dependency, so saturation is visible before errors appear.

• Synthetic checkouts: continuous end-to-end test orders (with test accounts and payment sandboxes) from several vantage points, measuring the real customer path.

• Money-trail metrics: ambiguous payments backlog, age of UNKNOWN items, refund backlog, reconciliation breaks.

• Runbooks and controls: pre-written, tested runbooks for each failure class, kill switches with scopes (global, region, cell), routing override controls, and a clear decision authority during the event. Changes are frozen except for approved mitigations.

• War room process: staffed shifts, a single incident commander, communication channels to payment partners and banks (who may also be struggling), and pre-agreed escalation with seller and logistics teams.

• Post-event review: capacity versus forecast, funnel losses and their causes, payment route performance, hot-SKU behaviour, and cost, feeding the next event's plan.

4.13 Putting It All Together

Before the event, deal prices, promo rules, and tax tables are precomputed and pushed to every cell's cache, capacity is pre-scaled and warmed, deals are staggered to avoid a single instant, and the whole stack is load-tested above forecast with failure injection. At the spike, the edge and CDN absorb browse traffic, priority classes reserve capacity for checkout, and customers land in their home cell. Cart operations are cheap and non-authoritative, checkout start computes the authoritative price from cached data and issues a signed price lock, and placement runs a short parallel critical path: verify the lock, reserve inventory (conditional decrements for cold SKUs, in-memory single-writer owners for hot SKUs), run a fast risk check with safe defaults, and create a payment intent routed across multiple PSPs by live health. Payment results flow through an inbox with idempotent processing, ambiguity is a first-class state, and late or duplicate confirmations resolve by re-reserving or refunding. The order is created exactly once with a unique constraint and a transactional outbox, then everything else (fulfilment, seller notification, invoicing, loyalty, notifications) is asynchronous, replayable, and allowed to lag. Cells, shuffle sharding, bulkheads, deadlines, retry budgets, and an ordered degradation ladder keep failures local and graceful, and live funnel monitoring, synthetic checkouts, and a war-room process keep the humans in control.

5. Database Design

5.1 SQL vs NoSQL

• Cart and checkout session store (distributed key-value, for example DynamoDB-style): single-key access by customer or checkout ID, high write rate, TTLs for abandoned sessions, horizontal scaling with no joins. Stored per cell with async cross-cell replication.

• Deal and price cache (in-memory, replicated per cell, versioned snapshots pushed from a central publisher): read-mostly, extremely hot, small in size. The source of truth is the catalogue and promotions systems, which are never in the request path.

• Inventory (two tiers):

  • Cold SKUs: a partitioned key-value store with conditional writes (available ≥ qty) per (sku, node).

  • Hot SKUs: in-memory single-writer owners with a replicated log and asynchronous durable projection, fenced by epochs.

• Order store (sharded relational or distributed SQL): orders, order lines, payments, and the outbox need transactions across a handful of rows per customer shard, unique constraints (idempotency), and strong consistency within a customer. Sharded by customer ID across hundreds of logical shards.

• Payments ledger (relational, append-only double-entry): financial correctness, auditability, and reconciliation.

• Event stream (Kafka or equivalent): order events, payment events, inventory events, and the outbox relay, partitioned by order or customer, with long retention for replay.

• Redis or in-memory caches: rate limits, idempotency fast paths, payment route health, offer-budget shard counters (with durable backing), hot-SKU admission state, per-customer read-your-writes caches.

• Search and read models (denormalised stores): "my orders" views, customer support lookups, built from the event stream.

• Object storage and a columnar analytics store: logs, funnel analytics, reconciliation files, archived orders.

• Secrets vault and HSM-backed key management: payment credentials, token signing keys (price lock), PII encryption.

5.2 Database Schema

-- Order shard (sharded by customer_id hash; one logical schema per shard)

orders (
  order_id          TEXT PRIMARY KEY,                 -- shard-prefixed, time-sortable
  customer_id       UUID NOT NULL,
  checkout_id       UUID NOT NULL,
  state             TEXT NOT NULL,                    -- PLACED|CONFIRMED|PACKED|SHIPPED|DELIVERED|CANCELLED|...
  state_version     BIGINT NOT NULL DEFAULT 0,
  currency          TEXT DEFAULT 'INR',
  items_total       NUMERIC(12,2), discount_total NUMERIC(12,2), shipping NUMERIC(10,2),
  tax_total         NUMERIC(12,2), payable NUMERIC(12,2),
  payment_mode      TEXT,                             -- UPI|CARD|NETBANKING|WALLET|EMI|COD
  address_snapshot  JSONB NOT NULL,
  price_lock_id     TEXT, rules_version TEXT,
  risk_decision     TEXT,                             -- ALLOW|STEP_UP|HOLD_REVIEW
  placed_at         TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (customer_id, checkout_id)                   -- exactly-once order creation
);
CREATE INDEX idx_orders_customer ON orders (customer_id, placed_at DESC);

order_lines (
  order_id TEXT REFERENCES orders(order_id), line_no INT, offer_id TEXT, sku TEXT, seller_id TEXT,
  qty INT, unit_price NUMERIC(12,2), discount_breakup JSONB, tax_breakup JSONB,
  fulfilment_node TEXT, reservation_id TEXT UNIQUE, line_state TEXT,
  PRIMARY KEY (order_id, line_no)
);

payments (
  payment_id UUID PRIMARY KEY, order_id TEXT, order_intent_id TEXT NOT NULL,
  method TEXT, psp TEXT, route TEXT, amount NUMERIC(12,2),
  state TEXT,                                         -- INITIATED|AWAITING_CUSTOMER|AUTHORIZING|AUTHORIZED|DECLINED|UNKNOWN|VOIDED|CAPTURED|REFUNDED
  psp_txn_id TEXT UNIQUE, idempotency_key TEXT UNIQUE, bank_offer_id TEXT, instant_discount NUMERIC(10,2),
  created_at TIMESTAMPTZ, resolved_at TIMESTAMPTZ
);

payment_inbox (                                       -- idempotent callback processing
  message_id UUID PRIMARY KEY, psp TEXT, psp_txn_id TEXT, status TEXT,
  business_key TEXT UNIQUE,                           -- psp|txn|status
  payload_hash TEXT, received_at TIMESTAMPTZ DEFAULT now(), processed BOOLEAN DEFAULT false
);

outbox (
  outbox_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, topic TEXT, key TEXT, payload JSONB,
  created_at TIMESTAMPTZ DEFAULT now(), published_at TIMESTAMPTZ
);

refunds (refund_id UUID PRIMARY KEY, payment_id UUID, reason TEXT, amount NUMERIC(12,2), state TEXT, initiated_at TIMESTAMPTZ, completed_at TIMESTAMPTZ);

ledger_entries (                                      -- append-only, balanced pairs
  entry_id BIGINT GENERATED ALWAYS AS IDENTITY, txn_id UUID NOT NULL, account TEXT NOT NULL,
  dr_cr CHAR(1), amount NUMERIC(14,2) CHECK (amount > 0), ref_payment_id UUID, created_at TIMESTAMPTZ DEFAULT now()
);

-- Checkout session (key-value; TTL ~30 min)
checkout:{checkout_id} → {
  customer_id, cart_id, cart_version, lines[], address_id, totals, rules_version,
  price_lock: { token_id, expires_at }, state, order_intent_id?, reservation_ids[], version
}

-- Inventory, cold SKU (key-value, conditional writes)
inv:{sku}:{node} → { available, reserved, version }
reservation:{reservation_id} → { sku, node, qty, checkout_id, expires_at, state }   // unique per (checkout, line)

-- Inventory, hot SKU (single-writer owner; state mirrored to a replicated log)
hot:{sku}:{shard} → { quota_shard, held, sold, live{reservation_id -> (qty, expiry)} }
  RESERVE(res_id, qty): idempotent on res_id; if quota_shard - held - sold >= qty -> held += qty; append to log; OK
                        else try BORROW from sibling shard (bounded, logged) else SOLD_OUT
  CONFIRM(res_id): held -= qty; sold += qty        RELEASE(res_id): held -= qty
  Invariant across shards: sum(held + sold) <= quota

-- Offer budget (sharded counter with durable backing)
offer_budget:{offer_id}:{k} → remaining_k        // k in 0..K-1; spend picks shard k by hash(order); borrow on empty shard
offer_redemption:{offer_id}:{customer_id} → count   // per-customer limit, conditional increment

Price-lock token (signed, stateless verification):

{ "tok": "pl_9f2…", "cust": "c_123", "checkout": "co_8f2…", "lines": [ { "offer": "o_1", "qty": 1, "price": 41980.00, "tax": 6403.39 } ],
  "discounts": [ { "type": "BANK_OFFER", "id": "bo_77", "amount": 3000.00 } ], "rules_version": "r_2025_10_04_0",
  "exp": "2025-10-04T00:14:05Z", "sig": "…" }

Event schema (stream, partitioned by order or customer):

{ "type": "ORDER_PLACED", "order_id": "404-…", "customer_id": "c_123", "payable": 38980.00, "payment_mode": "CARD",
  "lines": [ { "sku": "SKU123", "qty": 1, "node": "BLR-7" } ], "risk": "ALLOW", "ts": "2025-10-04T00:00:07.412Z", "event_id": "evt_…" }

Redis key schema:

HASH   rl:{customer|device|ip}:{class}:{window}  → counters
STRING idem:place:{customer}:{checkout_id}       → order_intent_id   (TTL 24 h; DB unique constraint is authoritative)
HASH   pay_route:{route}:{issuer}                → success_rate_1m, p95_ms, timeouts, circuit_state, inflight
HASH   hotgate:{sku}                             → concurrent_in_checkout, admit_rate, mode
STRING ryw:order:{customer}                      → recent order ids (TTL 60 s, read-your-writes cache)

6. Design Deep Dive

6.1 Why a 50x Spike Isn't Solved by 50x Capacity: Amplification, Tail Latency, and Retries

• The naive plan and why it fails. "Provision 50x more servers" fails for four reasons: stateful dependencies (databases, payment partners, seller systems) don't scale linearly or at all in a few hours, the cost of idle capacity is high, some dependencies are external and fixed, and the failure modes at 50x are not the same as at 1x (they are dominated by contention, retries, and tail latency).

• Call-graph amplification. A checkout start touches ~14 services. At 48K starts/sec that is ~670K internal calls/sec. If the call graph has a slow or flaky node, its latency propagates up. With p99 = 100 ms per call and 7 sequential critical calls, the end-to-end p99 approaches 700 ms, and any retry multiplies the load on the already slow node.

• Retry storms as a positive feedback loop. Dependency D slows by 30%. Callers time out at 80% of their budget and retry. Load on D rises 1.3-2x, so D slows further, and more callers time out. Within a minute the system is saturated. Counter-measures used together:

  • Retry budgets: at most ~10% extra calls from retries per client, tracked as a token bucket.

  • Deadline propagation: each downstream call receives the remaining budget and fails immediately if it can't meet it, so work for abandoned requests is never started.

  • Adaptive concurrency limits at each service (for example gradient or AIMD controllers that lower the concurrency limit as latency rises), so the service sheds excess requests quickly and keeps latency low for the admitted ones.

  • Exponential backoff with jitter on clients, and a single in-flight placement per checkout.

  • Circuit breakers with half-open probes to protect a recovering dependency.

• Shorten the critical path. The design does each of the following to reduce chain length: price lock (no recompute at placement), precomputed risk scores, cached delivery promises, in-memory promo rules, and asynchronous everything post-payment. The synchronous placement path is three parallel calls (reserve, risk, payment-intent), so the p99 is the max of three, not the sum of seven.

• Tiering turns an unbounded problem into a bounded one. Because every dependency has a classification and a fallback, the system has a defined behaviour when any dependency fails: it doesn't need all 14 services to be healthy to take an order. The target is that the order path depends only on the cart/session store, inventory, payment, and the order store.

• Stateful scaling happens by pre-sharding. Instead of scaling databases during the event, the order store is pre-split into hundreds of logical shards (mapped to fewer physical nodes), so adding capacity during the event is moving shards, not resharding data under load.

• Cells bound the damage of anything that still goes wrong. If a cell's dependency saturates, 1/N of customers are affected, the others proceed, and operations can drain and repair the one cell.

• Capacity numbers need a safety factor for forecast error. Marketing forecasts for festival traffic are often off by 30-50% in either direction. The 2x provisioning and 1.3-1.5x load-test targets, plus the degradation ladder, absorb that uncertainty.

6.2 Hot-SKU Inventory: Strict Where It Matters, Soft Where It Doesn't

• Two very different inventory problems. Cold SKUs (millions of them) have low contention, so a conditional decrement in a partitioned key-value store is enough. Hot SKUs (a few thousand at peak) have 100K+ attempts/sec on a single logical counter, which no per-row database update can sustain (~700 updates/sec per row with synchronous replication).

• The hot-SKU owner design. A single-writer in-memory owner processes reservation requests as tiny operations (a comparison and an increment). It batches requests for a few milliseconds, writes the batch decision to a replicated log (group commit), and acknowledges all requests in the batch. Throughput is limited by the log's append rate, not the counter, and batching makes it hundreds of thousands of operations/sec per owner.

• Splitting the quota without stranding units. For the very hottest SKUs, a single owner may still saturate its network or log. The quota is split into sub-quotas across several owners, and requests are spread by hash. The risk is "stranding": one shard runs out while another has units. A bounded borrow protocol lets an empty shard request a chunk from a sibling (a logged, idempotent transfer), preserving the global invariant sum(held + sold) ≤ quota. Near the end (when remaining units are few), the SKU collapses to a single owner to give exact, fair "last units" behaviour.

• Why reserve at placement, not at cart. With ~65% of checkouts abandoned and ~35% of placements failing payment, reserving at add-to-cart would make stock look gone while most holds never convert. Reserving at placement keeps holds proportional to real intent, with a TTL covering payment. The cost is that a customer may reach payment and find the unit gone, which is handled by failing placement cleanly ("this deal just sold out") before they pay.

• Hold TTL and recycling. Placement-time holds last ~10 minutes (shorter for hot deals, for example 4-5 minutes, to recycle stock faster), with a bounded extension when a payment is in progress. Explicit failures release immediately. Released units feed back to waiting customers, and the cart page shows a coarse "limited stock" band, not a live count, to avoid refresh storms.

• Durability and failover. The owner's state is rebuilt from the log plus the durable reservation records, leases are epoch-fenced, and after failover the owner fails closed until reconciled. A durable unique constraint on reservation_id and a CHECK (sold ≤ quota) in the projected store are the final backstops against oversell.

• Per-customer limits. Deal items carry per-customer quantity limits (for example 1-2 units), enforced inside the reservation operation using a per-customer counter, so a single user (or a bot network of accounts) can't capture a disproportionate share. Account-farm detection (shared cards, addresses, devices) runs in the async risk path, cancelling and re-releasing units.

• Soft inventory for ordinary items. For non-limited SKUs, a small, policy-controlled oversell is acceptable, because the supply chain often has slack and customers dislike "out of stock" more than a rare cancellation. The reservation is a conditional decrement with an allowed negative floor, and fulfilment resolves shortfalls (substitute from another node, delay the promise, or cancel with an apology and a coupon).

• Seller-side inventory. Marketplace sellers update their stock asynchronously. The platform applies safety stock buffers during the festival (shown stock is conservative) and penalises sellers with high cancellation rates, since inventory inaccuracy is a major source of festival-day cancellations.

6.3 Price Integrity, Offer Budgets, and Coupon Abuse

• What can go wrong. A customer sees ₹38,980 and pays ₹39,500. Or a bank offer's budget is exceeded by 20% because thousands of concurrent redemptions raced. Or a coupon intended once per customer is redeemed fifty times. Each is a financial or trust loss.

• Price lock gives consistency without recomputation. The signed token carries line prices, discounts, and rule versions. At placement, the orchestrator verifies the signature and expiry, and charges the locked total. It doesn't re-run pricing. If a deal expires mid-checkout, the lock still honours the price within the lock window and subject to inventory, which is a customer-friendly and operationally simple rule. The lock window is short enough (~15 minutes) to bound exposure.

• Offer budgets as sharded counters. A global bank-offer budget (for example ₹50 crore) is split into K shards (for example 64), each a counter with remaining. A redemption of ₹1,500 picks a shard by hash, performs a conditional decrement (remaining ≥ 1,500), and on an empty shard tries a bounded number of other shards. The sum across shards is the budget. This spreads 3,000 redemptions/sec over 64 counters (~47/sec each), which a key-value store handles comfortably.

• Consume at placement, not at checkout start. Reserving budget when the checkout page loads would drain budgets through abandoners. The page shows a coarse band (HIGH/LOW/EXHAUSTED), and the actual consumption happens atomically at placement. If the budget is exhausted at that instant, the customer is told and can proceed at the undiscounted price with explicit confirmation, or choose another offer.

• Tolerated overshoot with a hard ceiling. For ultra-hot offers, small overshoots (for example up to 1-2% of budget) can be accepted by policy to avoid heavy coordination, but a hard ceiling, enforced by the durable counter, prevents runaway. Finance sets the tolerance in advance.

• Per-customer and per-instrument limits. Offer terms ("once per card", "max 3 per customer") use conditional increments on keyed counters (customer, card fingerprint, device), with the card fingerprint (not the raw number) used for PCI scope reasons. These checks happen inside the placement transaction.

• Coupon abuse. Many accounts using one payment instrument or address to farm coupons is caught by graph analytics in the async risk path. Orders are held, cancelled with refunds, and the budget consumed is returned to the shards (idempotent credit keyed by order).

• Compensation correctness. If an order fails after budget consumption (payment failure, reservation failure), the compensation returns the budget by an idempotent credit keyed by (order_intent_id, offer_id), driven by the saga's retry queue, so a crash can't leak or double-return budget.

• Display consistency. The cart, checkout, payment page, confirmation, and invoice all use the same locked breakdown, generated once, stored on the order, and never recomputed, so the customer sees identical numbers everywhere, and disputes can be answered from the stored record.

6.4 Payments at Peak: Routing, Ambiguity, and COD

• The weakest link isn't ours. At 16K payment attempts/sec, a 10% failure rate means ~1,600 failed attempts per second and a lot of angry customers, mostly from issuer banks, 3-D Secure servers, and UPI rails the platform doesn't control. The design's job is to route around trouble, set honest expectations, and never lose money or orders.

• Health-based routing at multiple granularities. Route health is tracked per (route, issuer, method), because a bank outage looks like a platform-wide failure if tracked only per route. Sliding windows with minimum sample sizes and damped weight changes prevent oscillation, and a small exploration share keeps alternatives warm.

• Method steering. At peak, the checkout page can re-order or hide methods by live health (for example, show UPI first if net banking for a major bank is failing), and show customer-facing guidance ("Bank X cards are experiencing delays; try UPI"). This reduces failed attempts, not just retries.

• Tokenised cards and frictionless authentication. Network tokens and issuer-recognised devices increase frictionless 3-D Secure rates and reduce OTP load on issuer ACS servers, which are a common peak-time bottleneck.

• UPI specifics. UPI collect requests time out if the customer doesn't approve, UPI intent flows depend on the customer's app switching, and rail-level rate limits exist per PSP handle. The platform spreads UPI traffic across multiple PSP handles and banks, polls status with backoff, and uses callbacks idempotently.

• Ambiguity resolution. UNKNOWN is bounded: status polling (backoff, capped), inbox-deduplicated callbacks, and reconciliation against settlement files, with the order held in PENDING_CONFIRMATION and the customer informed. A pending-payment lock prevents a second payment on the same intent (with a warning override).

• Hold interplay for hot SKUs. The hot-SKU hold is short (4-5 minutes). If payment is UNKNOWN at expiry, the system doesn't release immediately when the provider reports progress: it extends once for ~60-90 seconds. A late success then follows the re-reserve-or-refund path, always resolved within a bounded time.

• Duplicate payments. Two taps, two tabs, or two methods produce duplicates. Idempotency on order_intent_id, a unique constraint on psp_txn_id, and reconciliation that voids or refunds surplus authorisations keep money consistent, and each case is visible to the customer.

• COD at festival scale. COD volume is large (~25-30% of orders). The risk gate uses customer history (COD refusals, returns), address and pincode risk, order value caps, and device signals, with step-up (OTP or a confirmation call) for borderline cases. High-risk customers are shown "COD unavailable, pay online" messaging. Since a refused COD order wastes logistics cost, preventive checks at placement are cheaper than post-hoc losses.

• Offers funded by banks. Instant discounts depend on bank-side eligibility and funding. The orchestrator records the offer, the funding split, and the settlement obligations in the ledger, so finance can reconcile with banks after the event.

• Money-trail discipline. An independent reconciliation job matches orders, payments, PSP records, bank settlements, and refunds, classifies breaks (paid without order, order without payment, duplicate charges, stuck refunds), assigns owners and ages, and escalates. During the event it runs continuously, because ambiguity piles up fastest at peak.

6.5 Exactly-Once Orders, Sagas, and Safe Retries Under Chaos

• Where duplicates come from. Customers double-tap, mobile apps retry on flaky networks, gateways resend callbacks, workers crash mid-step, queues redeliver, and operators replay events after a fix. The system assumes every message can arrive twice and every step can be interrupted.

• Idempotency at each layer:

  • Client to API: Idempotency-Key = {customer}:{checkout_id}, stored with the first outcome, so a retry returns the original response.

  • Reservation: unique reservation_id per (checkout, line), so replays can't double-reserve.

  • Payment: unique idempotency_key per intent and unique psp_txn_id, with an inbox keyed by (psp, txn, status).

  • Order creation: UNIQUE (customer_id, checkout_id) on orders, which is the final guard.

  • Events: consumers key effects by (order_id, event_type) so replays don't double-invoice or double-notify.

• Why the database constraint is the real guard. Caches and fast-path idempotency stores are optimisations that can fail or expire. The unique constraints in the durable stores guarantee that even if every upstream check fails, a second order for the same checkout can't exist.

• The saga's recoverability. The orchestrator's state is persisted per transition, so a recovery worker can scan for stuck checkouts (for example RESERVED with no payment progress past the TTL, or PAID with no order) and drive them forward or compensate. Compensations (release reservation, void authorisation, return offer budget) are themselves idempotent and retried from a queue, never fire-and-forget.

• The "paid but no order" invariant. A payment AUTHORIZED state creates an obligation: the order must be created or the money returned. A reconciler looks for authorised payments without orders older than a few minutes and either creates the order (if the reservation is still valid) or refunds. This closes the gap in which a crash between payment confirmation and order creation could otherwise leave money in limbo.

• Outbox and consumer idempotency. The order write and its events are committed together, the relay publishes at-least-once, and consumers deduplicate. This is what lets downstream systems lag, restart, or replay without corrupting results.

• Ordering where it matters. Events for one order are partitioned together, so consumers see a coherent sequence, and state transitions are guarded by version compare-and-set, so a late CONFIRMED event can't overwrite a CANCELLED state.

• Cancellation races. Customers cancel within minutes of ordering. A cancel arriving while fulfilment planning is processing is resolved by the order state machine (cancel wins until the order passes a point of no return), and compensating events release stock, void or refund, and notify the seller, each idempotently.

• Testing the ugly paths. Fault-injection tests deliberately kill orchestrator nodes mid-saga, duplicate callbacks, delay events, drop acknowledgements, and expire reservations during payment, and the pass criterion is invariants (no duplicate orders, no paid-without-order, no stock leaks), not just availability.

6.6 Capacity Engineering, Cells, and Running the Event

• Forecasting. Traffic forecasts combine prior-year curves, marketing calendars (deal schedules, push-notification times, influencer campaigns, bank partnerships), early-access registrations, and growth. The forecast is expressed per deal wave and per cell, not as a single peak number, and has explicit confidence bands. The plan provisions to the upper band for tier 0, and relies on degradation for anything beyond it.

• Cell design details.

  • A cell is sized to a tested capacity (for example 1,000-1,500 orders/sec), with the whole stack replicated (cart, orchestrator, pricing cache, reservation proxies, payment orchestrator, order shards).

  • Global shared services are minimised and classified. Hot-SKU owners, payment routing health, and offer budgets are the main cross-cell components, so they have their own isolation and capacity plans, and they run with static fallbacks (for example local payment-route weights if the global health service is stale).

  • Customers stick to their home cell. A failover cell has the replicated data it needs and headroom to take a failed neighbour's load (at most ~1/N extra each, designed in).

• Shuffle sharding for shared dependencies. Each cell (or customer group) maps to a randomly assigned subset of instances of a shared service (for example 4 of 40 risk-service instances), so a bad request pattern or a failing instance affects only the few cells sharing that exact subset.

• Deployment hygiene. A change freeze starts days before the event, with an exception process for fixes and a canary-then-cell-by-cell rollout with automatic rollback. Config changes (flags, routing weights) are treated like deployments, with validation and gradual rollout. Feature flags default to the safe state, and kill switches are tested in advance.

• Pre-event checklist. Caches primed, deal snapshots verified in every cell, capacity verified by load test, payment partners' capacity confirmed and their on-call contacts arranged, DNS and CDN configurations verified, certificates and secrets with long validity, dashboards and alerts reviewed, runbooks rehearsed, synthetic probes running, and customer-support readiness (scripts for payment pending, order delays).

• During the event.

  • A war room with clear roles (incident commander, payments lead, inventory lead, edge lead, communications).

  • A small number of golden signals displayed prominently: successful orders/sec, checkout p99, payment success rate by method, ambiguous-payment backlog, error budget burn, and saturation of the tier 0 dependencies.

  • Actions follow runbooks with pre-approved levers: shift payment weights, disable a method, enable a deal gate, shed a tier, drain a cell, flip soft inventory.

  • Communication to customers is honest and consistent (app banners, status page), because an honest "high demand, we're confirming your payment" message prevents the double-ordering that makes things worse.

• After the spike. Don't scale in prematurely, since a second wave often follows. Drain downstream backlogs (invoices, notifications, fulfilment planning), run the full reconciliation, handle ambiguous payments and refunds, review cancellations from oversell, and then scale down gradually.

• Learning loop. Every event produces a review: forecast versus actual by wave, funnel loss analysis, dependency incidents and their blast radius, hot-SKU behaviour, payment route performance by issuer, cost per order, and customer-impact measures (cancellations, ambiguous-payment duration, support contacts). The findings change the capacity model, the deal calendar, the degradation ladder, and the architecture for the next event.

• Cost control. The most expensive resources are stateful tiers and external payment fees. Using stateless cells for elasticity, caching aggressively, shedding Tier 2, and compressing the event window through staggered deal starts reduce peak provisioning. Capacity is released promptly after the event, and reserved capacity commitments (where used) are sized to the baseline plus a share of peak.

Summary

• A 50x spike isn't solved by 50x servers. The binding constraints are stateful dependencies, external payment partners, call-graph amplification (~14 calls per checkout, ~670K internal calls/sec), tail latency, and retry feedback loops, so the design focuses on shortening the critical path, bounding retries, tiering dependencies, and containing blast radius.

• Tier the platform and protect checkout. Tier 0 (cart, price lock, inventory reservation, payment, order creation) gets reserved capacity, tight timeouts, and no optional dependencies. Tier 1 has cached or default fallbacks, and Tier 2 (recommendations, ads, reviews) is shed first, with priority classes enforced at the edge and in every service.

• Do work before the event and after the order, not during checkout. Deal prices, promo rules, tax tables, delivery estimates, and risk scores are precomputed and pushed into every cell's cache, a signed price lock removes re-pricing from placement, and invoices, seller notification, fulfilment planning, loyalty, and notifications run from events that are allowed to lag.

• Placement is a short, parallel, resumable saga. Verify the lock, reserve inventory, run a fast risk check with a safe default, and create a payment intent in parallel, with deadlines propagated downward, per-dependency concurrency limits, circuit breakers, and persisted state so any node can resume or compensate after a crash.

• Inventory is dual-mode. Cold SKUs use conditional decrements in a partitioned store, hot SKUs use single-writer in-memory owners with group-commit replicated logs, sub-quotas with a bounded borrow protocol, epoch-fenced failover that fails closed, per-customer limits inside the reservation, and reservation at placement (not cart) with short TTLs and one bounded extension, plus soft, bounded oversell for ordinary items.

• Price and offer integrity are enforced mechanically. Offer budgets are sharded counters consumed at placement (the page shows only a coarse band), per-customer and per-instrument limits use conditional increments, compensation returns budget idempotently, and the locked breakdown is stored on the order and never recomputed.

• Payments are routed, steered, and reconciled. Health-based routing per route, issuer, and method with damped weights and an exploration share, method re-ordering and customer messaging during bank issues, tokenised cards and frictionless authentication, UPI spread across handles, UNKNOWN as a bounded state resolved by polling, inbox-deduplicated callbacks, and settlement reconciliation, late-success re-reserve-or-refund, duplicate refunds, and a risk-gated COD path.

• Orders are created exactly once. UNIQUE (customer_id, checkout_id), idempotency keys at every layer, a transactional outbox, idempotent consumers, version-guarded state machines, and a "paid but no order" reconciler, with fault-injection tests that assert invariants (no duplicate orders, no paid-without-order, no stock leaks).

• Cells, shuffle sharding, and bulkheads contain failures. Independent cells with tested capacity and home-cell stickiness, shuffle-sharded shared dependencies, pre-split order shards so scaling is moving shards, and cell-by-cell deployments, with a change freeze during the event.

• Capacity is pre-scaled, warmed, and verified. Stateless tiers at ~2x forecast peak and stateful tiers at ~1.5x with verified headroom, caches primed, runtimes and connection pools warmed, load tests at 1.3-1.5x forecast with failure injection, staggered deal starts to turn one vertical spike into several smaller steps, and deal-level admission gates for the few ultra-hot SKUs.

• Degradation is ordered and automated. Stale prices and stock bands, Tier 2 shed, offers disabled with clear messaging, cached delivery promises, unhealthy payment methods hidden, deal gates enabled, soft inventory for ordinary items, low-priority consumers paused, and as a last resort a static high-demand page for part of one cell, never a cascading failure.

• Operate it as an event. Live funnel and dependency-saturation dashboards, synthetic checkouts, money-trail metrics, tested kill switches and routing levers, a staffed war room with a single incident commander, honest customer messaging, delayed scale-in, full reconciliation afterwards, and a review that feeds the next year's forecast and architecture.