IRCTC Tatkal: A System Where 5 Million Users Try to Book Train Tickets at Exactly 10 AM

IRCTC Tatkal: A System Where 5 Million Users Try to Book Train Tickets at Exactly 10 AM

1. Clarifying Requirements

1.1 Functional Requirements

• At a fixed, publicly known time (10:00 for AC classes, 11:00 for non-AC, one day before departure), a quota of last-minute "Tatkal" tickets opens for each train. Millions of users attempt to book within the same minute, and the system must sell each seat exactly once.

• Users can log in, search trains, choose a train/class/date, enter passenger details, solve a verification challenge, pay, and receive a confirmed ticket (PNR) or a clear failure, all inside a very short window.

• The system enforces business rules on Tatkal bookings:

  • a cap on passengers per ticket (for example 4),

  • a cap on tickets per user per day,

  • ID requirements for passengers,

  • no concession fares,

  • restrictions on agents during the first minutes of the window,

  • quota-specific waitlist caps.

• The system maintains a seat inventory per train, journey date, class, and quota, split by journey segment, since a seat sold from A to C blocks that seat for overlapping legs.

• The system holds inventory while the user pays for a bounded time, releases it if payment fails or times out, and converts it to a confirmed booking on successful payment.

• The system integrates with multiple payment channels (UPI, cards, net banking, wallets), each with its own latency and failure behaviour.

• The system handles the case where money is debited but the booking is not completed, with automatic confirmation if inventory allows, otherwise a prompt automatic refund.

• Users can save passenger lists and payment preferences beforehand so that the booking path is as short as possible in the critical minute.

• The system generates confirmed or waitlisted tickets, e-ticket and SMS/email delivery, and later supports status checks and cancellation under Tatkal's rules.

• Operations teams can pre-scale, pause or throttle, change admission rates, monitor funnel stages in real time, and extend or hold the window in exceptional circumstances.

• The system provides audit, reconciliation, and dispute handling, and produces the evidence needed to investigate agents or bots.

1.2 Non-Functional Requirements

• Demand is hundreds of times supply. Five million users compete for perhaps a few hundred thousand seats, and on popular routes demand exceeds supply by a factor of 100 or more. Most users will not get a ticket regardless of engineering, so the design goals are fairness, honesty, and not collapsing, not satisfying everyone.

• Hard correctness on inventory. A seat is sold at most once. Overselling Tatkal (a confirmed ticket for a berth that does not exist) is a severe failure. Under-selling (seats left unsold while users fail) is a serious but lesser failure.

• The window is brief and the critical path is long. Within the first 60-120 seconds, popular trains sell out. The path from login to confirmation involves many sequential steps, so each step's reliability and speed compounds, and a 3-second delay at one step can decide who gets a seat.

• Fairness and bot resistance. Automated tools and booking agents with scripts, multiple accounts, and fast hardware can capture a disproportionate share. The system should favour genuine individual users, limit multi-account abuse, and make fairness defensible.

• Availability under extreme, synchronised load. The system must degrade gracefully instead of falling over, since a crash at 10:00:05 means millions of failed attempts, public outrage, and no clear winner. Predictable slowness with clear communication is better than a crash.

• Payment is the weakest link. Payment gateways and banks cannot absorb the peak. Many failures happen at this stage, and "payment deducted, ticket not booked" is the most common complaint category. The design must treat payment as an unreliable, slow, external dependency.

• Latency matters differently at each stage. Search and login must be fast, but the inventory hold must be bounded in time (users get a few minutes to pay), and confirmation must be exactly-once and durable.

• Idempotency and exactly-once effects. Retries by users, browsers, and gateways must not create duplicate bookings or duplicate charges.

• Integrity of the process. Public confidence depends on the sense that the system is fair and that the same rules apply to everyone, with auditable evidence.

• Regulatory and operational constraints. Identity verification (for example Aadhaar-linked accounts), ID checks at travel, data protection, and tight coordination with the passenger reservation core, which also serves unreserved/general bookings, charting, and cancellations.

2. Back-of-the-Envelope Estimation

Demand at the opening moment:

• 5M users are present at 10:00:00, a large fraction already logged in. Assume 70% (3.5M) hold an authenticated session before the window opens, and 1.5M log in at or after 10:00.

• Each active user fires about 6-10 requests in the first minute (search, train list, availability, book click, passenger submit, captcha, payment init), and many refresh repeatedly. That gives roughly 30-50M requests in the first 60 seconds, or ~500-800K requests/sec averaged over the minute, with the first 10 seconds running well above that (1M+ requests/sec at the edge).

• Most of this load is read-like (static assets, availability and train lists) or repeated failing attempts. Only a small share ever reaches the inventory.

The funnel and where it narrows:

• Of 5M users, about 80% attempt a booking (4M). Of those:

  • ~30-40% fail at the login/session or captcha step (timeouts, wrong captcha),

  • ~50% abandon or fail at passenger details (slow pages, validation errors),

  • perhaps 5-10% (250-500K) reach the inventory hold attempt,

  • and only the seats that exist can be held successfully: perhaps 150-250K actual holds across all trains in the first minutes.

• The inventory service therefore sees ~50-100K hold attempts/sec at its peak, which is a fraction of edge traffic but far more concentrated.

Supply:

• Assume about 250-300K Tatkal berths are bookable in the 10:00 window across all trains (AC classes), but demand is concentrated: the top ~200 train-class combinations attract roughly 60-70% of all booking attempts.

• A hot train-class might have ~100-250 Tatkal seats and receive 3,000-8,000 hold attempts per second at the peak. This is the single most important fact for the inventory design: a very small, very hot counter.

Payment load:

• 150-250K successful holds in the first minutes means 150-250K users heading to payment within roughly 2-4 minutes, or ~1,000-2,000 payment initiations/sec at peak.

• A payment gateway or a bank's UPI/net banking rail may reliably sustain a few hundred transactions per second per route, with timeouts rising above that. With a ~3-5 minute hold and 20-40% of payments failing or timing out at peak, tens of thousands of holds are released and retried during the first 10 minutes, and thousands end up in an ambiguous "pending" state.

Hold accounting:

• With a 3-minute hold at ~1,500 holds/sec, concurrent live holds reach roughly 200-300K. Since supply in the hot trains is a few hundred seats each, a hold that ties up a seat for 3 minutes blocks that seat from the next 3,000 users, so hold duration directly trades off fairness, throughput, and utilisation.

• If 30% of holds fail to convert, the effective throughput of seats-per-minute drops by the same factor, and seats recycle back to the pool several times in the first 10 minutes, which creates a second, smaller wave of success for persistent users.

Data volumes:

• A booking record with passengers, fare breakup, and payment references is ~3-5 KB. 250K Tatkal bookings/day is ~1 GB, trivial. Event and audit logs for the opening minute (billions of request records, mostly from failed attempts) reach 100s of GB, so edge and gateway logging must be sampled, summarised, and tiered.

• Session and token state for 5M users at ~1-2 KB is ~5-10 GB in memory, which is modest.

Connection and bandwidth:

• 5M concurrent sessions, each opening a few connections, requires tens of millions of connection-lifetimes through the edge in the first minute, and page weight (even at ~200 KB per page, 30M page views = 6 TB in a minute) means static assets must come from a CDN and the dynamic payloads must be tiny.

3. Core APIs

Pre-opening readiness

POST /v1/session/prepare
Body: { device_token, client_attestation }
Response: { session_id, csrf_token, server_time_offset_ms, window_opens_at: "2025-03-14T10:00:00+05:30",
            prefetch_bundle_version: "pb_8832" }

Waiting-room entry / admission

POST /v1/queue/enter
Body: { session_id, intent: { train_no, journey_date, class, quota: "TQ" } }
Response: { queue_ticket: "qt_...", position_hint: 48210, status: "WAITING" | "ADMITTED",
            admission_token: null | "adm_...", expires_in_s: 120, poll_after_ms: 1500 }

GET /v1/queue/status?ticket=qt_...
Response: { status: "WAITING" | "ADMITTED" | "EXPIRED", admission_token, estimated_wait_s }

Availability (cacheable, eventually consistent)

GET /v1/availability?train=12951&date=2025-03-15&class=3A&quota=TQ
Response: { status_band: "AVAILABLE" | "FEW" | "WL" | "SOLD_OUT", approx_count: 12, as_of: "10:00:03.2", ttl_ms: 500 }

Begin booking: take an inventory hold

POST /v1/bookings/hold
Headers: { Authorization, X-Admission-Token, Idempotency-Key: "u123:12951:20250315:3A:attempt1" }
Body: {
  train_no, journey_date, class, quota: "TQ", from: "NDLS", to: "BCT",
  passengers: [ { name, age, gender, berth_pref, id_type, id_ref } ],
  challenge: { type: "CAPTCHA", token, answer_ref }
}
Response: { hold_id: "h_...", status: "HELD" | "WAITLIST_OFFERED" | "SOLD_OUT" | "REJECTED",
            hold_expires_at, fare: { total, breakup }, payment_options: ["UPI","CARD","NB"] }

Payment initiation and result

POST /v1/payments/initiate
Body: { hold_id, method: "UPI", vpa?, bank?, idempotency_key }
Response: { payment_id, redirect: {...} | collect_request: {...}, state: "INITIATED", deadline: "..." }

GET /v1/bookings/{hold_id}/status
Response: { state: "HELD"|"PAYMENT_PENDING"|"CONFIRMED"|"WAITLISTED"|"FAILED"|"REFUND_INITIATED"|"EXPIRED",
            pnr: null | "2451987663", message, next_check_after_ms }

Callbacks and reconciliation (internal)

POST /v1/internal/payments/callback         (signed, idempotent)
Body: { payment_id, psp_ref, status: "SUCCESS"|"FAILED"|"PENDING", amount, bank_event_time }

GET  /v1/internal/reconciliation/{psp}/{date}

Operations and control

PUT  /v1/admin/admission-rate     Body: { train_class: "*", per_second: 4000, mode: "ADAPTIVE" }
POST /v1/admin/windows/{id}/hold-window   Body: { hold_seconds: 180 }
POST /v1/admin/kill-switch        Body: { feature: "WAITLIST", enabled: false }
GET  /v1/admin/funnel/live        → { stage_counts, error_rates, p95_by_stage, psp_health }

4. High-Level Design

4.1 Design Principle: Move Work Before 10:00, Shed Work at the Edge, Protect the Inventory Core

Components needed:

The system's load profile is a funnel where 99% of requests will fail or be abandoned, and the design succeeds only if those failures are cheap and early, while the small number of requests that reach the inventory are fast, fair, and correct. Three principles shape everything:

• Do everything possible before the window opens. Authenticate early, pre-load the booking UI and passenger master data, validate passenger details in advance, and warm caches and connections.

• Make the failing majority cheap. Serve static content from the CDN, absorb retries at the edge, return compact responses, and never let a rejected request touch the inventory store.

• Meter admission to what the downstream can handle. A waiting room and admission tokens convert an unbounded flood into a controlled stream, rate-matched to the inventory service and the payment channels.

The architecture has five layers: edge and CDN, session and bot-defence, waiting room and admission control, booking core (inventory, holds, confirmation), and payment orchestration, with the reservation system of record behind the booking core.

4.2 Pre-Opening Phase: Shrinking the Critical Minute

Components needed:

Early login and session warming. Users are encouraged (through in-app prompts and UX) to log in 10-15 minutes before the window. Session establishment, OTP, CAPTCHA at login, and device binding all happen before 10:00, so the login stampede is spread across many minutes instead of landing at the opening second. Sessions are long-lived tokens validated locally at the edge, with no per-request database calls.

Prefetch bundle. The client downloads a versioned bundle (static booking UI, station codes, train timetable, fare tables, validation rules) ahead of time and caches it, so that at 10:00 only tiny API calls are needed. Time synchronisation is also done up front, with a server time offset (server_time_offset_ms) so client countdowns and button enablement align with server time without the clients all polling the clock.

Saved passenger master list and pre-validated IDs. Users maintain a master list of passengers (names, ages, ID types and numbers). The system pre-validates these (format checks, ID verification where possible), so the passenger-details step in the critical minute is a selection, not typing. Pre-validation removes a large source of failure (typos and rejected IDs) and shaves tens of seconds off a booking.

Pre-authorised payment methods. Users can pre-register UPI handles or a wallet balance (a prepaid wallet is the fastest and most reliable payment path since it is internal to the platform) and pre-link cards with tokenised credentials, so the payment step needs minimal input. A prepaid wallet balance also bypasses external gateway dependency for those users, which relieves the weakest link (section 4.6).

Pre-computed state. Train availability bands and fare quotes for the opening window are precomputed and cached. Connection pools to the booking core, payment services, and caches are warmed, and the capacity for the 10:00 window is pre-scaled well in advance, since reactive autoscaling cannot respond within seconds.

Scheduled opening, not instantaneous gating. The "Book" button enables at a server-synchronised moment. To avoid a perfectly aligned thundering herd from clients that all fire at exactly 10:00:00.000, clients add a small random jitter (tens to a couple of hundred milliseconds) to their first request, which flattens the very first spike without disadvantaging anyone in expectation (section 6.2 explains why this is fair).

4.3 Edge, Bot Defence, and Identity: Making Automation Expensive and Fairness Defensible

Components needed:

Edge protection. A CDN and global load balancer terminate TLS, serve static assets, apply WAF and DDoS rules, and enforce coarse rate limits per IP, ASN, and device. Connection-level limits prevent a single client from holding thousands of sockets.

Layered bot defence.

• Device and client attestation: the official app uses platform integrity attestation, and the web client uses browser integrity and proof-of-work challenges. Attested clients are favoured, but attestation can be bypassed, so it is a signal, not a gate.

• Behavioural signals: request timing regularity, absence of human-like input timing (keystroke and pointer dynamics on the web), headless-browser fingerprints, and replayed sessions.

• Challenge placement: CAPTCHAs at login (before the window) filter bulk account use cheaply, and a booking-time challenge (a lightweight or invisible challenge, escalated to an interactive one for suspicious clients) protects the hold step. Challenges are bound to the session and the specific booking attempt, with short expiry, so a solved challenge can't be reused or farmed out for later.

• Account-level controls: one active booking session per account, limits on tickets per user per day, limits on distinct accounts per device and per payment instrument, and phone and ID verification at registration, which raises the cost of mass account creation.

• Agent restrictions: registered agents are subject to rules (for example, barred from booking during the first minutes of the Tatkal window), enforced by account type, with detection for agents using personal accounts to evade the restriction (shared payment instruments, repeated passenger-detail patterns, device clusters).

Uniform-looking responses. Rejections for suspected bots look the same as capacity-related rejections ("high demand, try again"), so adversaries cannot easily tune their tooling against detection signals, and legitimate users aren't accused.

Identity linkage. Verified identity (for example, an Aadhaar-linked account) lets the system apply per-person limits across accounts, supports dispute resolution, and enables the policy that Tatkal tickets be booked only by verified users, accepting privacy and exclusion trade-offs that the policy and legal teams own.

Defence economics. The goal is not to stop every bot but to make large-scale automation costly enough (account creation costs, device costs, CAPTCHA solving costs, blocked IP ranges) that it no longer dominates, and to ensure that hold-time and per-user limits cap the damage of any single actor.

4.4 Waiting Room and Admission Control: Turning a Flood into a Metered Stream

Components needed:

Why a waiting room. Even with cheap edge handling, allowing 4-5M users to hit the booking core simultaneously would overwhelm it. A virtual waiting room at the edge accepts everyone into a lightweight queue, issues a queue ticket, and admits users to the booking path in controlled batches by issuing signed, short-lived admission tokens. Booking APIs reject any request lacking a valid token (validated at the edge from a signature, with no backend call).

What the queue protects:

• The inventory core, so hold attempts per second stay within what the counters and database can process.

• The payment channels, since admission rate bounds payment initiations (a user admitted now will hit payment in about 1-2 minutes).

• The user experience, since 5M users on spinning pages with timeouts is worse than a queue with honest messaging.

Adaptive admission rate. The admission rate is not a fixed number. A controller adjusts it using live signals:

• hold-service latency and error rate,

• inventory-store load,

• the hold-to-payment conversion rate and payment success rate (if payment channels are failing, admitting more users only creates more failing holds),

• remaining seat counts in the hot trains (when a train is nearly sold out, admission for that train can be reduced or the user informed).

Per-train (per-intent) queues. A single global queue is wasteful. A user queued for a sold-out train shouldn't wait behind others, and a user wanting an available train shouldn't be blocked by hot-train demand. The queue can be sharded by train-class-date (users declare intent when entering), with each shard admitted according to that shard's inventory and hot-spot status. When a shard sells out, its queue is released immediately with a clear "sold out" message, and users can re-enter for alternative trains.

Fair ordering within the queue. Pure arrival-order FIFO rewards the lowest latency and the fastest tooling. The design adopts a randomised-within-window policy: everyone who enters within the first N seconds (for example the first 30-60 seconds around 10:00) is assigned a random position, and later arrivals queue behind in arrival order. This neutralises millisecond-level advantages without punishing people who arrive at 10:00:02, and is simple to explain and audit (section 6.2).

Honest communication. The waiting room shows position estimates, an explanation of what will happen, and, crucially, realistic odds ("high demand: most users will not get a ticket for this train; consider alternatives"). Honesty reduces frantic refreshing (which is load) and builds trust.

Token properties. Admission tokens are bound to the session, device fingerprint, and intent (train-class-date), expire in about 1-2 minutes if unused, and are single-use for the hold step. They cannot be traded or replayed, and their issuance is logged for audit.

Behaviour when the queue itself is overwhelmed. The waiting-room tier is designed to be extremely cheap per user (a static page, a long-poll or server-sent-events channel, or a client that polls every 1-3 seconds with jitter), implemented at the CDN/edge or in a very lightweight service with state in a distributed store, so even millions of waiting users cost little. If it still saturates, it sheds by serving a static "high demand" page, never by passing load to the booking core.

4.5 Inventory, Holds, and Confirmation: The Correctness Core

Components needed:

Inventory model. Inventory is tracked per (train, journey date, class, quota) with segment awareness. For availability, the usual approach is to model each coach's berths per origin-destination leg and compute whether a requested journey has a free berth for all intervening segments. For the Tatkal hot path, the booking core uses a precomputed availability structure per train-class-quota that yields, in O(1), whether n seats are available for the requested segment pair, and a berth allocator assigns specific berths at confirmation.

Count-based hold at the front, berth assignment at the back. Two-phase handling keeps the hot path fast:

1. Hold reserves a count of seats (for example 2) for the requested segment, as an atomic decrement on an in-memory counter, guaranteeing that a seat will exist at confirmation.

2. Confirmation (after payment) runs the berth allocator, which assigns concrete berths and coaches (respecting family-together and lower-berth preferences for seniors where rules apply), within the reserved count, so allocation work isn't done for the huge number of holds that never convert.

Atomic hold with expiry. A hold is created through a single atomic operation (for example a Redis Lua script or an in-memory shard owner executing a compare-and-decrement) that: checks the user's per-day and per-train limits and idempotency key, checks the available count for the segment pair, decrements, records the hold with a TTL (the hold window, for example 3 minutes), and returns a hold_id. If insufficient, it returns SOLD_OUT or offers a waitlist position. The operation is linearisable per inventory shard, which is what prevents overselling.

Sharding the hot counter without overselling. A hot train-class with ~200 seats and 5K attempts/sec concentrates all attempts on one key. Options:

• A single-owner shard handling the hot key in memory on one core can easily process tens of thousands of operations per second, because each operation is a few memory reads and writes, so for most trains a single-threaded owner per key is sufficient and simplest, avoiding distributed coordination.

• For the extreme hot keys, the owner node is dedicated and isolated (its own CPU, no co-tenant keys), requests are batched (process 50-100 requests per cycle), and the admission controller keeps the attempt rate within what the owner handles.

• Seat pre-splitting (dividing a train's remaining seats among several sub-counters) can raise throughput but complicates "last seats" fairness and requires rebalancing, so it is used only if a single owner can't keep up (section 6.1).

Durability and recovery of in-memory state. The authoritative record is durable: every hold, release, and confirmation is appended to a replicated log (or written to a database) before it is acknowledged, or the in-memory owner is replicated synchronously to a standby. After a failure, the standby replays the log and takes over, with fencing tokens to prevent a stale owner from continuing (section 6.5). Counters can always be rebuilt from the log plus the confirmed-bookings table.

Hold lifecycle.

• HELD (TTL running) → PAYMENT_PENDING (payment initiated, TTL possibly extended once, see below) → CONFIRMED (payment success, berths allocated, PNR issued) or EXPIRED/RELEASED (timeout, failure, user cancel).

• A release atomically increments the counter and makes the seats available again. A sweeper process also scans for expired holds (TTL expiry events and a periodic sweep), so a lost expiry event never strands seats.

• Hold-window policy. A short hold (for example 2-3 minutes) cycles seats faster and increases fairness, while too short a window causes failures for users on slow payment methods. Policy can differ by payment method: wallet users get a short window, and UPI/net-banking users get a slightly longer one, and the window is adjustable by operations in real time (section 3).

Limits enforcement. Per-user per-day Tatkal limits, per-train duplicate-attempt checks, and per-PNR passenger caps are checked inside or immediately before the atomic hold, using the same shard owner or a fast per-user store, so a single user cannot hold seats across many trains and starve others (a cap on concurrent live holds per user, typically one, is critical).

Waitlist handling. When confirmed seats are exhausted, Tatkal may allow a limited waitlist. A waitlist entry is also a counted resource (a capped counter), and confirmation order follows hold order, with automatic promotion on cancellation per the rules. Because Tatkal waitlists carry refund consequences, the user is shown a clear message before they commit to paying for a waitlisted ticket.

Confirmation and PNR issuance. On payment success, the confirmation service: verifies the hold is still valid (or handles the late-success case, section 4.6), runs the berth allocator, writes the booking to the system of record, generates the PNR, emits events for e-ticket and SMS, and marks the hold converted. Confirmation is idempotent on hold_id, so duplicate payment callbacks produce one booking.

Interaction with the reservation core. The existing passenger reservation system may be the system of record for berths and charting. The Tatkal inventory layer acts as a front-line allocation service with its quota partitioned from the general pool (a Tatkal quota released at 10:00), and synchronises confirmations to the core through a reliable, idempotent channel, with reconciliation ensuring that counts match. The core is never exposed to the opening-minute traffic.

4.6 Payment Orchestration: Designing Around the Unreliable, Slow, External Dependency

Components needed:

Payment is the dominant failure source. Gateways and banks have latency in seconds, timeouts, and capacity limits far below the booking demand, and failures at this step both waste held seats and generate the most user distress. The orchestration layer is designed on the assumption that payment results are slow, duplicated, delayed, or lost.

Channel strategy.

• Prepaid wallet (platform-internal): instant, reliable, no external dependency, so it is promoted heavily for Tatkal. A wallet debit and booking confirmation can happen in one local transaction.

• UPI: high success but a customer-approval step and bank-dependent latency, with a collect-request or intent flow. Timeouts commonly resolve to "pending", needing status polling.

• Cards and net banking: redirect flows, higher abandonment and failure, and strict bank capacity limits.

• Multiple PSPs and routes with health-based routing and weighted traffic, so a degraded gateway or bank loses traffic automatically and capacity is pooled.

Payment state machine and ambiguity. States: INITIATED → AWAITING_USER → PENDING_AT_BANK → SUCCESS | FAILED, with a PENDING-unknown state that is resolved by active status inquiry and eventually by the PSP's settlement file. The orchestrator never treats a timeout as a failure: it is "unknown" until a definitive answer arrives.

Hold interplay: the central trade-off.

• If the hold expires while payment is pending, releasing the seat immediately risks a later "success" that has no seat (a paid-for ticket with nothing to confirm), while holding seats indefinitely for pending payments starves other users.

• Policy: the hold is extended once, briefly (for example +60-90 seconds) when a payment is actively in progress (a bank has acknowledged the request), and then released. If a success arrives after release, the late-success path runs (below).

Late success after seat release.

1. Check whether seats are still available for the segment (the released seat may still be unsold, since most hot seats are reallocated quickly, but not always).

2. If available, atomically re-hold and confirm at the original fare, preserving the user's priority.

3. If not available, automatically refund the full amount through the PSP's refund mechanism, with a clear message, and record a deterministic reason code. Offer a waitlist option where rules allow, only with explicit consent.
   This policy prevents "money taken, nothing delivered, no resolution" from persisting, which is the largest complaint class.

Payment-first versus hold-first. The standard design holds seats first, then collects payment. An alternative for UPI is to collect the payment authorisation first and finalise the booking only if a seat is available, but this leads to many refunds. The design keeps hold-first (users get certainty of a reserved seat while paying) and mitigates its downside (stranded holds) with short windows, one-hold-per-user limits, and wallet promotion.

Idempotency and exactly-once confirmation. Each payment has an idempotency key tied to the hold. Callbacks are deduplicated by PSP transaction ID, and the transition to CONFIRMED uses compare-and-set on the hold state. A user retrying payment on the same hold cannot be charged twice, since a second payment attempt on a hold already PAYMENT_PENDING is either rejected or attached to the same payment record, and a second successful charge is automatically refunded.

Reconciliation. The platform reconciles three records: its own payment and booking records, the PSPs' transaction and settlement files, and the reservation core's confirmed bookings. Differences produce categories (paid-not-booked, booked-not-paid, duplicate charges), each with a defined automated or manual resolution path, an owner, and an age-based escalation (the same discipline used in the earlier payment designs in this series).

Backpressure to admission. The payment success rate and PSP latency feed back into the admission controller (section 4.4), because admitting users into a payment layer that is failing only converts demand into stranded holds. Healthy payment capacity effectively sets the system's sustainable booking throughput.

4.7 Capacity, Degradation, and Resilience

Components needed:

Static stability and pre-scaling. Everything needed for the window is provisioned in advance (edge capacity, waiting room, booking core, payment orchestration), with no reliance on autoscaling or control-plane operations during the first minutes. Capacity is tested with realistic replays (and load tests that model the funnel, including retries and abandonment).

Layered degradation ladder (ordered):

1. Serve slightly stale availability bands (hundreds of milliseconds to a few seconds old) from the cache, instead of live counts.

2. Disable non-essential features (promotions, recommendations, rich UI, optional add-ons such as insurance, meal selection), returning minimal payloads.

3. Reduce the admission rate and lengthen waiting-room polling intervals.

4. Switch the booking page to a lightweight, single-step form for hot trains.

5. Prefer wallet payments (surface only wallet and UPI during peak if card and net-banking channels are failing).

6. Close admission for sold-out trains immediately and free those users.

7. As a last resort, show a static high-demand page with a retry-after, protecting the core from collapse.

Isolation (bulkheads). Hot trains are isolated on dedicated inventory owners, so contention on 200 hot keys doesn't affect the thousands of normal trains. The waiting room, booking core, and payment orchestration scale and fail independently. Partner/agent API traffic uses a separate rate budget and lane from consumer traffic.

Multi-AZ and regional resilience. Active-active across availability zones, with synchronous replication of inventory state within a region (a standby owner per hot key), a warm secondary region for disaster recovery, and clear fencing to prevent two owners accepting writes for the same inventory (section 6.5). Because inventory correctness requires a single writer per key, cross-region active-active inventory is not used for the Tatkal path: a region failover is a controlled promotion, not a split.

Retry discipline. Clients use exponential backoff with jitter. The edge enforces per-session retry limits and returns Retry-After hints. Idempotency keys make retries safe, and the waiting room replaces manual refreshing with a managed wait.

Failure scenarios planned for:

• Inventory owner crash: standby promotion with log replay and fencing, with in-flight holds either preserved (if replicated) or cleanly invalidated with the user shown a retry message.

• Redis/cache loss: counters are rebuilt from the durable log and confirmed-bookings table, admission pauses during recovery, and the system fails closed (no holds) rather than risk overselling.

• Payment gateway degradation: automatic rerouting, admission throttling, and wallet promotion.

• Reservation core slowness: confirmations queue durably, users see PAYMENT_PENDING → CONFIRMED shortly after with an honest "confirming" state, and the hold stays valid.

• Waiting-room outage: fall back to a static queue page with a randomised retry timer, not an open door to the core.

4.8 Observability, Operations, and Governance

Components needed:

• Real-time funnel dashboards: counts and p95 latency per stage (edge, queue, hold, payment, confirmation), conversion rates between stages, error and timeout rates per payment route, admission rate and queue depth per train shard, hold-expiry rates, and seat recycling counts.

• Control levers with guardrails: admission rate, hold window, feature kill switches, payment route weights, per-train throttles, all behind approvals and audited, with automated suggestions (for example "UPI route B timeouts at 28%, recommend reducing weight").

• Fairness and abuse analytics: distribution of booking times and success by client type, concentration of successes by payment instrument, device, or IP range, repeated passenger-detail patterns, and anomalies after the window, feeding enforcement (account suspension, cancellation of fraudulent bookings) with documented appeal paths.

• Auditability: every admission token, hold, payment, and confirmation is traceable with timestamps, to answer "why did this user not get a ticket" and "was this booking legitimate" with evidence.

• Game days and pre-event rehearsals: full-scale rehearsals (including the waiting room, payment failure injection, and owner failover), run during low-demand periods and before festival peaks, and a post-window review each day comparing predicted and actual funnel behaviour.

• Communication: status banners, accurate expectations, and rapid, clear messaging during incidents, since the perception of fairness and honesty is part of the product.

4.9 Putting It All Together

Before 10:00, users log in, download a prefetch bundle, and select pre-validated passengers and payment methods, spreading identity load and shortening the critical path. At 10:00, clients fire with small random jitter into a CDN and edge layer that absorbs the flood, rejects bots with layered, uniform-looking defences, and places users into a per-train waiting room with randomised-within-window ordering. An adaptive admission controller releases signed, short-lived tokens at a rate that the inventory core and payment channels can sustain. Admitted users perform an atomic, idempotent seat-count hold on a single-owner, durably logged counter, then pay through an orchestrated, multi-route, wallet-friendly payment layer that treats every external answer as possibly slow or ambiguous, extends holds minimally, resolves late successes by re-holding or refunding automatically, and reconciles against PSP and reservation-core records. Confirmation allocates berths, issues the PNR idempotently, and notifies the user. Throughout, degradation is ordered, honest, and fail-closed on correctness, and the system is judged by no overselling, no collapse, no lost money, and visible fairness.

5. Database Design

5.1 SQL vs NoSQL

• In-memory inventory owners (custom service, or Redis with Lua) for the hot counters. Hold and release are a handful of memory operations that must be linearisable per key and handle tens of thousands of operations per second on one key. A single-writer, in-memory structure is the right fit, backed by a durable log.

• A replicated, ordered log (Kafka, or a Raft-based log) for inventory events. Every hold, release, and confirmation is appended before acknowledgement (or replicated to a hot standby), enabling recovery, audit, and downstream consumers.

• PostgreSQL (or the reservation system's relational database) for authoritative bookings, passengers, payments, and refunds. These need transactions and constraints (unique PNR, unique idempotency_key, unique hold_id per confirmation) and strong audit. Write volume is modest in absolute terms (hundreds of thousands of bookings per day), so a well-provisioned relational database with partitioning by journey date suffices, shielded by the queue and holds.

• Redis cluster for sessions, admission tokens, waiting-room state, rate limits, and per-user limit counters. Fast, TTL-native, and rebuildable.

• A distributed key-value store (or Redis with persistence) for the waiting-room queue state where very high write rates and simple structures dominate. Sharded by intent (train-class-date).

• CDN and edge compute for static assets, the prefetch bundle, bot challenges, and admission-token verification (signature checks at the edge).

• Object storage and a columnar analytics store for request logs, funnel analytics, fairness analysis, and reconciliation files, with sampling at the edge to control volume.

• A secrets vault and HSM-backed keys for token signing, PSP credentials, and encryption of sensitive passenger data.

5.2 Database Schema

-- PostgreSQL (authoritative booking and payment records)

users (
  user_id UUID PRIMARY KEY, phone_hash TEXT, id_verification_status TEXT,
  user_type TEXT,                       -- INDIVIDUAL | AGENT
  risk_score REAL, created_at TIMESTAMPTZ
);

passenger_master (
  passenger_id UUID PRIMARY KEY, user_id UUID REFERENCES users(user_id),
  name TEXT, age INT, gender TEXT, id_type TEXT, id_ref_enc BYTEA, validated BOOLEAN
);

inventory_buckets (                     -- durable snapshot; hot path lives in memory
  train_no TEXT, journey_date DATE, class TEXT, quota TEXT,
  total_seats INT NOT NULL, confirmed INT NOT NULL DEFAULT 0, waitlist_cap INT, waitlisted INT DEFAULT 0,
  version BIGINT NOT NULL,
  PRIMARY KEY (train_no, journey_date, class, quota)
);

holds (
  hold_id UUID PRIMARY KEY, user_id UUID, train_no TEXT, journey_date DATE, class TEXT, quota TEXT,
  from_station TEXT, to_station TEXT, seat_count INT NOT NULL,
  state TEXT NOT NULL,                  -- HELD|PAYMENT_PENDING|CONFIRMED|EXPIRED|RELEASED
  idempotency_key TEXT UNIQUE NOT NULL,
  created_at TIMESTAMPTZ, expires_at TIMESTAMPTZ, extended BOOLEAN DEFAULT false,
  admission_token_id TEXT, state_version BIGINT NOT NULL DEFAULT 0
);
CREATE INDEX idx_holds_expiry ON holds (expires_at) WHERE state IN ('HELD','PAYMENT_PENDING');
CREATE INDEX idx_holds_user_active ON holds (user_id) WHERE state IN ('HELD','PAYMENT_PENDING');

bookings (
  pnr TEXT PRIMARY KEY, hold_id UUID UNIQUE REFERENCES holds(hold_id),   -- one booking per hold
  user_id UUID, train_no TEXT, journey_date DATE, class TEXT, quota TEXT,
  status TEXT,                          -- CONFIRMED|WAITLISTED|CANCELLED
  fare_total NUMERIC(10,2), booked_at TIMESTAMPTZ
);

booking_passengers (
  pnr TEXT REFERENCES bookings(pnr), seq INT, name TEXT, age INT, gender TEXT,
  coach TEXT, berth INT, berth_type TEXT, status TEXT, id_type TEXT, id_ref_enc BYTEA,
  PRIMARY KEY (pnr, seq)
);

payments (
  payment_id UUID PRIMARY KEY, hold_id UUID REFERENCES holds(hold_id),
  psp TEXT, route TEXT, method TEXT, amount NUMERIC(10,2),
  state TEXT,                           -- INITIATED|AWAITING_USER|PENDING_AT_BANK|SUCCESS|FAILED|UNKNOWN|REFUNDED
  psp_ref TEXT UNIQUE, idempotency_key TEXT UNIQUE,
  initiated_at TIMESTAMPTZ, resolved_at TIMESTAMPTZ
);

refunds (
  refund_id UUID PRIMARY KEY, payment_id UUID REFERENCES payments(payment_id),
  reason TEXT,                          -- LATE_SUCCESS_NO_SEAT|DUPLICATE_CHARGE|HOLD_EXPIRED|SYSTEM
  amount NUMERIC(10,2), state TEXT, initiated_at TIMESTAMPTZ, completed_at TIMESTAMPTZ
);

admission_log (
  token_id TEXT PRIMARY KEY, session_id TEXT, user_id UUID, intent TEXT,
  issued_at TIMESTAMPTZ, used_at TIMESTAMPTZ, outcome TEXT
);

reconciliation_exceptions (
  exception_id UUID PRIMARY KEY, kind TEXT,    -- PAID_NOT_BOOKED|BOOKED_NOT_PAID|DUPLICATE|ORPHAN
  payment_id UUID, hold_id UUID, detected_at TIMESTAMPTZ, owner TEXT, resolved_at TIMESTAMPTZ
);

Inventory owner (in-memory) structures and hold script (conceptual):

shard key: (train_no, journey_date, class, quota)
state:     available_by_segment_pair[...] | counters, live_holds{hold_id -> (count, segment, expiry)}
HOLD(idem_key, user_id, count, segment):
   if idem_key seen -> return prior result                      -- idempotency
   if user_live_holds[user_id] >= 1 -> REJECT (limit)
   if available(segment) >= count -> decrement; record hold; append to log; return HELD(hold_id, expires_at)
   elif waitlisted < waitlist_cap -> WAITLIST_OFFERED
   else -> SOLD_OUT
RELEASE(hold_id) -> increment; remove hold; append to log
CONFIRM(hold_id) -> assign berths from reserved count; append to log

Redis key schema (non-inventory):

STRING sess:{session_id}                 → { user_id, device_hash, risk, auth_time }
STRING adm:{token_id}                    → { session, intent, exp }     (TTL ~120 s, single use)
HASH   queue:{train|date|class}:meta     → admitted_count, rate, sold_out flag
ZSET   queue:{train|date|class}:waiting  → score = assigned position, member = queue_ticket
STRING ulim:{user_id}:{date}             → tickets booked today
HASH   rl:{ip|device|account}:{window}   → request counts, failure counts
HASH   psp_health:{route}                → success_rate_1m, p95_latency, circuit_state

Event log schema (Kafka, partitioned by inventory shard key):

{ "seq": 8821934, "shard": "12951|2025-03-15|3A|TQ", "type": "HOLD_CREATED|HOLD_RELEASED|CONFIRMED|WAITLISTED",
  "hold_id": "h_9f2", "user_id": "u123", "count": 2, "segment": "NDLS-BCT", "ts": "2025-03-14T10:00:03.481Z" }

6. Design Deep Dive

6.1 The Hot Counter: Throughput, Correctness, and Why a Single Owner Usually Wins

• The shape of the problem. A hot train-class has ~100-250 Tatkal seats, receives thousands of hold attempts per second, and sells out in under a minute. Almost all attempts fail, and the few that succeed must be exactly right.

• Why a single-writer in-memory owner is attractive. Each hold attempt is a few comparisons and an increment, which takes microseconds. One core can process hundreds of thousands of such operations/sec. Even 8K attempts/sec for a train is trivial for one owner, so the owner is not the bottleneck, the surrounding network, serialisation, and logging are. A single owner gives linearisability with no locks, no distributed transactions, and trivial reasoning about oversell safety.

• Where the real cost lives. The expensive parts are request parsing, authentication, durable log append (batched), and replication to the standby. These are handled by batching (group commit: collect requests for a few milliseconds, process the batch against memory, append once, acknowledge all), which amortises durability cost across hundreds of requests.

• Segment-aware availability. For multi-leg journeys, the structure tracks availability per segment pair efficiently (for example per-leg free-berth counts with range-minimum queries, or precomputed bitmaps per coach), so a hold for A→C checks legs A-B and B-C together. This is more work than a single counter but still microseconds, and hot trains mostly have a single dominant origin-destination pair.

• When a single owner isn't enough. If benchmarks show that one owner can't keep up (very large inventories or slow log appends), options include sub-counter splitting (divide remaining seats among K sub-owners, route a request to a random sub-owner, and rebalance when a sub-owner empties), accepting a small fairness skew near sell-out. A "last seats" protocol collapses back to a single owner when the remaining seats drop below a threshold, to prevent a user being told "sold out" while another sub-owner holds spare seats.

• Fail closed. If the owner or its durable log is unavailable, holds are refused, never optimistically granted. Under-selling for a minute during failover is acceptable, while overselling is not.

• Admission control keeps the owner in its comfort zone. The waiting room limits the attempt rate per shard to a level the owner and its log can sustain, so latency stays flat instead of climbing into timeouts, which would trigger retries and worsen load.

6.2 Fairness: What "First" Means When Network Latency Dominates

• The unfairness of raw arrival order. At 5M users, differences of tens of milliseconds in network distance, device speed, or scripted submission decide who is "first". Pure FIFO gives systematic advantages to users near data centres and to automation, and the first 100 ms can contain more attempts than there are seats.

• Randomised-within-window admission. All requests arriving in a window around the opening (for example 10:00:00 to 10:00:30, plus client jitter) are treated as simultaneous and assigned random queue positions, using an auditable random function (for example a keyed hash of (ticket_id, window_seed) with the seed committed in advance and revealed later). Each user has an equal chance, independent of milliseconds. Users arriving after the window enter behind in arrival order.

• Client jitter and why it's fair in expectation. Clients add a random delay (for example 0-200 ms) before their first request. Because the randomised window dwarfs this delay, jitter doesn't affect odds, and it flattens the edge-level spike that would otherwise make the first milliseconds the busiest of the whole system.

• Per-intent queues and the "choose wisely" problem. Since queues are per train, users who pick the hottest train face worse odds than those picking an alternative. The system helps by showing honest demand indicators ("very high demand") and alternatives, but it does not secretly balance users across trains, which would be both unfair and confusing.

• Defending against queue gaming. One queue entry per verified account per intent, session binding, device limits, and detection of multiple accounts per person. Queue positions are not transferable, and admission tokens are bound to the session and device.

• Fair outcomes versus fair opportunities. The design guarantees equal opportunity (random order among simultaneous arrivals, bot limits, per-user caps), and does not promise equal outcomes. Public communication states this plainly, and aggregate fairness statistics (distribution of success across regions, device types, and account ages) are published or audited internally to demonstrate that the process isn't favouring any group.

• Edge cases: users with slow connections get a longer admission-token validity (within limits) so that network slowness doesn't forfeit a granted slot, and accessibility accommodations (alternate challenge types) are provided so that fairness measures don't exclude users with disabilities.

6.3 The Hold Window: Throughput, Fairness, and Seat Recycling

• The trade-off. A seat held for 3 minutes can't serve anyone else during that time. If 30% of holds never convert (failed payments, abandonment), long holds waste seat-time, while very short holds cause legitimate users on slow payment methods to lose their seats mid-payment.

• Quantifying it. With ~200 seats and a 30% non-conversion rate, a 3-minute hold means that roughly 60 seats are tied up for 3 minutes each by users who will ultimately fail. Those seats are recycled to the next waiting user only after expiry, so hold duration sets the recycle cadence, and shorter holds produce more recycling waves and more total successful sales over the first 10 minutes.

• Differentiated windows. The hold window is tuned by payment method: wallet (fast, near-deterministic) gets about 60-90 seconds, UPI gets 2-3 minutes with a bounded extension when the bank is actively processing, and net banking/cards get slightly longer because of redirect overhead. The user sees a visible countdown so there is no surprise.

• Early release signals. When a user explicitly cancels, closes the payment flow, or a payment definitively fails (a clear decline), the hold is released immediately, not at timeout. The client sends an explicit abandon signal where possible (best-effort, since the tab may simply close).

• One live hold per user. This is the most effective anti-hoarding rule: a user (or bot) can't pin seats on multiple trains while deciding. Attempting a second hold releases the first (with a warning), so a user cannot starve others across trains.

• Extension policy. A single extension is granted only when the payment is in a state that signals progress (a bank-acknowledged pending state), never on user request, so the extension can't be abused as an indefinite reservation.

• Recycling fairness. When a seat returns to the pool, it should go to the next eligible user in queue, not to whoever's request happens to hit the owner first. The admission controller, aware of releases (through the event log), admits the next waiting users for that shard in order, keeping late-stage fairness intact. In practice the admission rate for a nearly sold-out shard is tied to the release rate, so queued users aren't all admitted to compete for a handful of recycled seats.

6.4 Payment Ambiguity: Money Debited, No Ticket

• Where it arises. The bank debits the account, but the callback is delayed or lost, the user's browser closes before returning, the hold expires during the pending period, or the PSP reports PENDING and later SUCCESS after the booking has been released. These are the cases that generate the loudest complaints, and they are structurally unavoidable with external payment rails.

• Principle: every payment ends in either a confirmed ticket or a refund, and never silently in neither. The orchestrator tracks every payment to a final state and has a deadline (hours, not days) for resolution of UNKNOWN states, driven by status inquiry polling and settlement-file reconciliation.

• Resolution paths (as in section 4.6):

  • Success within the hold: confirm.

  • Success after the hold expired: attempt a fresh atomic hold for the same user and segment; if seats exist, confirm at the original fare, otherwise refund automatically.

  • Failure: release the hold (if still held) and inform the user.

  • Unknown: keep the payment record open, inform the user honestly ("we are confirming with your bank, no action needed, you will receive a ticket or an automatic refund by [time]"), and prevent them from paying again for the same journey without a warning (to avoid duplicate charges).

• Duplicate charge prevention. A second payment attempt for a hold that already has an unresolved payment is blocked with an explanation, and if a second charge nonetheless succeeds (the user paid twice through two tabs), the second is refunded automatically and logged.

• Late-success fairness question. Re-holding a seat for a late success could "jump the queue" over users who were waiting for that seat. The policy is deliberate: the late-success user already paid within the valid process, so priority is given only if the seat is still unallocated at that moment, within a short grace period (for example a few minutes after release), after which the refund path applies. This bounds the unfairness to waiting users, and the rule is published.

• Refund speed and communication. Refunds follow the PSP's rules and timelines (instant for wallets, typically T+1 to T+7 for banks). Users get an immediate in-app status ("refund initiated, reference number, expected timeline") and proactive updates, which cuts support contact dramatically.

• Operational dashboards for the money trail. Counts of payments in each state, age of UNKNOWN items, refund backlog, and PSP-specific unknown rates, with alerts when the unknown rate climbs (an early sign of a PSP incident), so operations can reroute traffic or reduce admission before the problem grows.

6.5 Correctness Under Failure: Owner Failover, Fencing, and Recovery

• Single-writer safety. The system's oversell protection depends on exactly one owner processing each inventory key at any moment. A network partition or a long pause can create a zombie owner that believes it is still the owner while a standby has been promoted.

• Fencing tokens and leases. Ownership is granted by a lease with a monotonically increasing epoch. Every log append carries the epoch, and the durable log (or the confirmed-bookings store) rejects appends from a stale epoch. A zombie's writes fail, and it stops serving once it cannot renew its lease (self-fencing), so only the newest epoch's writes become durable.

• Synchronous replication before acknowledgement. A hold is acknowledged only after it is durable on the log (or replicated to a standby quorum). A user told HELD therefore has a hold that survives an owner crash. The cost is a few milliseconds of latency, absorbed by batching.

• Recovery procedure. On promotion, the new owner rebuilds the in-memory state by replaying the log from the last snapshot, reconciles live holds against their expiry times (holds whose TTL elapsed during failover are released, with a small safety margin to avoid releasing seats for a user mid-payment), and only then begins serving. During recovery, the shard's admission is paused, queued users see a short "finalising" message, and the system refuses holds rather than guessing.

• Counter integrity checks. Continuous invariant checks compare confirmed + live_holds + available == total for every shard from the log and the bookings table. Any violation pages operations and freezes that shard, since an invariant break means either a bug or a lost event. A periodic full reconciliation after the window confirms that confirmed tickets exactly equal the number of seats consumed.

• Idempotent confirmation. The confirmation path is a transaction in the booking database with a unique constraint on hold_id in bookings, so duplicate payment callbacks, retries, and replayed events cannot produce two PNRs for one hold.

• Reservation-core synchronisation failures. If the system of record is slow or down, confirmations are queued durably, the user sees CONFIRMING, and the hold remains protected until the core acknowledges. If a core-side rejection occurs (a rare inconsistency), a compensating path refunds and explains.

• Testing the failure modes. Chaos tests kill owners mid-batch, partition the network, delay log appends, duplicate callbacks, and expire holds during failover, with automated invariant checking as the pass criterion. Each rehearsal checks that oversell is impossible, not merely unlikely.

6.6 Bots, Agents, and the Economics of Abuse

• Why this system is a target. Tatkal tickets have a resale value, and agents who capture inventory can charge a premium, so there is a strong, well-funded incentive to automate: pre-filled passenger data, script-driven form submission, parallel accounts, solving CAPTCHAs through services, and fast infrastructure close to the data centre.

• No single control suffices, so raise the cost per ticket. The defences compound:

  • Verified identity and account age: limited Tatkal bookings per verified person, new accounts restricted, phone and ID verification.

  • Booking-time challenges bound to session and attempt, with escalation for suspicious clients, and challenge difficulty tied to risk, so legitimate users usually see a lighter challenge.

  • One live hold per user and one queue entry per account per intent, which cap the benefit of speed.

  • Per-device and per-payment-instrument limits (many accounts sharing one device, card, or UPI handle is a strong signal).

  • Passenger-detail pattern limits (the same passenger appearing across many accounts' bookings).

  • Behavioural scoring across the funnel, with delayed enforcement (accept the booking but flag it, then cancel fraudulent bookings after investigation), so that attackers can't iterate in real time against visible block decisions.

  • Agent rules enforced by account type during the restricted window, plus detection of agents operating through personal accounts.

• Detection after the fact is as important as prevention. Post-window analytics identify clusters (shared devices, payment instruments, naming patterns, booking cadence), enabling cancellation and account bans with an appeal process. The knowledge that fraudulent bookings may be cancelled and accounts banned reduces incentives beyond what real-time blocking can.

• Collateral damage and trust. Aggressive defences produce false positives that hurt genuine users. The system uses graduated responses (extra challenge before block), clear appeal channels, and metrics on false-positive rates (for example success rates of long-tenured accounts), accepting that some automation will get through to avoid excluding real users.

• Arms race hygiene. Detection rules and thresholds change frequently, signals are never revealed in error messages, rules are updated without client releases where possible, and red-team exercises (internal teams attempting to automate bookings) measure the real cost to an attacker and identify weak points before adversaries do.

• Legal and policy alignment. Terms of service, restrictions on automation, penalties for violators, and cooperation with law enforcement for organised ticket-touting networks complement technical measures, since the incentive is economic and legal deterrents change the calculation.

6.7 Making the Failing Majority Cheap: Protecting the System From Its Own Users

• Retry amplification is the main self-inflicted risk. When responses slow down, users click again, scripts retry, and mobile apps auto-retry, multiplying load exactly when capacity is scarcest. A 2x slowdown can cause a 3-5x traffic increase, tipping the system into collapse.

• Defences against retry storms:

  • The waiting room replaces manual refreshing with a managed wait, which is the largest single reduction in redundant requests.

  • Server-directed pacing: responses include Retry-After and poll_after_ms, clients obey them with jitter, and the edge penalises clients that ignore them.

  • Request coalescing and caching at the edge for availability, with sub-second TTLs, so identical requests collapse.

  • Idempotency keys ensure that retries that do reach the core cost almost nothing (a lookup of the prior result).

  • Fast, honest rejection. Rejections are returned at the edge in microseconds without touching the core, and responses are tiny.

• Cheap sold-out handling. When a train-class sells out, the shard's flag propagates to the edge within a second, so further attempts are answered "sold out" at the edge and the waiting room releases all users for that shard with alternatives, removing thousands of requests per second from the core. Seats recycled later are announced through the same mechanism to waiting users, not to the open crowd.

• Protecting the long tail of other trains. Hot-train load must not degrade normal trains' bookings, so each hot shard has dedicated capacity and admission, and per-shard queues keep their pressure local.

• Capacity of the edge itself. The CDN and waiting-room tiers are sized for the full 5M plus retries, and fall back to static pages if they are overwhelmed, because a static "high demand" page is a graceful degradation while a failing dynamic backend is not.

• Cost view. Because the vast majority of requests are served at the edge or the waiting room, the expensive tier (inventory, payment, reservation core) sees only a few percent of the traffic. This is the economic essence of the design: spend capacity where correctness is needed and keep everything else cheap, cacheable, and disposable.

Summary

• Supply is tiny and demand is enormous, so the goals are correctness, fairness, and not collapsing. Five million users compete for a few hundred thousand seats, hot trains see thousands of attempts per second on a few hundred seats, and the design is judged by no overselling, no crash, no lost money, and defensible fairness, not by satisfying everyone.

• The load is a funnel, so most work is moved before 10:00 and most failures are made cheap. Early login, session warming, a prefetched UI bundle, pre-validated passenger lists, and pre-linked payment methods shorten the critical path, a CDN and edge layer absorb static and repeated traffic, and only a few percent of requests ever reach inventory or payment.

• A waiting room with adaptive admission turns a flood into a metered stream. Per-train queues, signed single-use admission tokens bound to session and intent, and an admission rate tuned from inventory health and payment success keep the correctness core within capacity, and sold-out shards release their queues immediately with honest messaging.

• Fairness is randomised within the opening window. Users arriving in the first seconds get random positions (with an auditable seed), client jitter flattens the first spike, one live hold and one queue entry per verified account limit hoarding, and the process promises equal opportunity, not equal outcomes, with fairness statistics audited.

• Inventory is a single-writer, in-memory, durably logged counter per train-class-quota. Atomic, idempotent seat-count holds with per-user limits, group-commit batching, standby replication, fencing epochs, and fail-closed behaviour give linearisable, oversell-proof allocation, with berth assignment deferred to confirmation so allocation work isn't wasted on holds that never convert.

• The hold window is a deliberate throughput-fairness lever. Short, payment-method-aware holds with visible countdowns, a single bounded extension when a bank is actively processing, immediate release on explicit failure or abandonment, and admission tied to release rates keep seats recycling quickly to the next waiting users.

• Payment is treated as slow, ambiguous, and external. Wallet promotion, multi-PSP health-based routing, a state machine with an explicit UNKNOWN state, status inquiry and settlement reconciliation, a late-success path that re-holds if a seat is still free or refunds automatically, duplicate-charge protection, and payment health feeding back into admission mean every payment ends in a ticket or a refund.

• Abuse is fought by raising the cost per ticket. Verified identity, session-bound challenges, device and instrument limits, passenger-pattern detection, agent-window rules, delayed enforcement with post-window cancellations and bans, uniform-looking rejections, and red-team testing, balanced against graduated responses and appeals to limit false positives.

• Degradation is ordered, static, and honest. Pre-scaled capacity, stale-tolerant availability, trimmed features, reduced admission, single-step booking forms, wallet-first payment, and finally a static high-demand page, with bulkheads that isolate hot trains from the rest, client backoff with jitter, and Retry-After pacing to prevent retry storms.

• Correctness is verified continuously. Invariants (confirmed + live holds + available = total), idempotent confirmation keyed on hold_id, three-way reconciliation of platform, PSP, and reservation-core records, chaos tests that kill owners and delay logs, and full-scale rehearsals ensure that failure modes produce refusals and refunds, never oversold berths or silently lost payments.