Design a Ticket Reservation System for Flash Concert Sales

Assumptions: Buyers are authenticated before checkout, the payment provider supports idempotency keys and authorization followed by capture, and the eight-minute hold is a deadline to complete a purchase—not a guarantee that a payment started at the last second will succeed. Safety takes precedence over availability during an ambiguous regional failure.

Architecture: Put a CDN and bot protection in front of a virtual waiting room, API gateway, and stateless seat-map, reservation, order, and payment services. Run services across multiple availability zones in the primary region, with a warm standby in the second region; autoscale sale-facing services. Use one transactional database as the authority for seat state, holds, orders, and per-user limits. Replicate its committed state synchronously to the standby region. Publish database outbox events to a message bus for notifications and cache updates. Serve seat maps from a cache, with updates pushed to clients; displayed availability is advisory, not a promise. Keep the critical write path independent of the cache and message bus.

Holds and purchases: Give each event-seat pair one database row. A hold request locks the buyer’s per-event limit record and the requested seat rows in a consistent order, checks that the buyer will hold at most six seats, and atomically claims seats that are available or whose holds have expired. Store hold owner, token, and expiry using database time. Return conflicts without partial success unless the client explicitly accepts fewer seats. Neither cache entries nor a timer decide ownership: every competing hold and purchase checks the authoritative row in a transaction. A worker expires holds for prompt map updates, while reads and writes also treat expired holds as available, so a failed worker cannot strand seats.

At checkout, create a durable order attempt tied to the hold and an idempotency key. After payment authorization is definitively successful, atomically change the seats to SOLD and the order to CONFIRMED only if every seat still belongs to that hold and has not expired. Commit the confirmation and its outbox event together. Capture payment after confirmation; retry failed captures and reconcile them operationally. If confirmation loses the race to expiry, void or refund the authorization. A sold seat never becomes available again through hold expiry. Client retries use the same attempt key and retrieve the existing outcome rather than creating another purchase.

Opening-sale traffic and fairness: Admit users through the waiting room at a rate bounded by measured database and payment capacity, not by the number of web servers. Randomize positions among users present before the announced opening, then admit later arrivals in arrival order. Issue signed, short-lived admission tokens; enforce per-account purchase limits, bot challenges, and rate limits without letting repeated refreshes improve position. Show queue position estimates and shed nonessential traffic. Cache static seat-map geometry at the CDN and coalesce availability updates; cap concurrent holds and checkouts. Admission slows when lock contention, payment backlog, or error rates rise. A user admitted to browse gets no seat priority until a hold transaction commits.

Payment uncertainty: Never hold a database lock across a provider call. Use the provider’s idempotency key for retries and webhooks as evidence to reconcile timed-out or unclear authorizations; an HTTP timeout is not a decline. Query the provider before another attempt, and keep uncertain attempts visibly pending. Do not confirm an order without a definitive authorization and a successful seat transaction. If the eight-minute deadline passes first, the seats can be reclaimed; any later authorization is voided or refunded, not turned into a confirmed order. Alert on unresolved attempts and reconcile against provider settlement records. Disable new checkouts if reconciliation is unsafe rather than risk duplicate charges or sales.

Reliability and targets: Multi-zone instances, load balancers, and database failover handle ordinary node or zone failures. Stateless services can restart; a failed cache falls back to rate-limited authoritative reads; a failed bus delays maps and notifications but not purchases. Database transactions, durable logs, synchronous cross-region replication before acknowledging confirmations, backups, restore tests, and outbox replay protect confirmed orders. Keep the standby warm and test failover under sale load. Promotion requires fencing the old writer and verifying replicated progress: two regions alone cannot safely guarantee both immediate writes during a network partition and zero lost confirmations. Pause purchases if fencing or replication is uncertain. Measure 99.95% purchase-flow availability over a defined monthly window, excluding neither provider failures nor admission failures, and budget capacity and failover time against it; a sufficiently long regional partition may breach the target rather than permit overselling.

Trade-offs: Use a transactional seat database rather than Redis locks: higher write cost buys auditable, atomic ownership and orders. Use one writer with fenced failover rather than active-active seat writes: less geographic write availability, but no conflicting owners. Use cached, slightly stale maps rather than database reads for every viewer: conflicts can occur at selection, but a 500,000-user arrival does not overwhelm the authority. Use bounded admission rather than unlimited autoscaling: some users wait, but admitted checkouts remain usable and infrastructure need not be provisioned for the entire surge. Release expired holds despite uncertain payment rather than extending them indefinitely: inventory returns on time, at the cost of occasional voids or refunds.

Sale dashboard and alerts: Track queue depth, wait-time distribution and admission rate; API availability and latency; hold success, conflicts, expiry lag, per-user-limit failures and database lock waits; checkout conversion and order-confirmation latency; payment timeouts, pending-attempt age, authorization-to-confirmation mismatches, refunds and capture failures. Alert immediately on any duplicate-seat invariant violation or lost confirmed order; page on replication lag, failed fencing, low database capacity, rising 5xx rates, excessive payment uncertainty, and expiry backlog. Track cache staleness and outbox lag separately from authoritative purchase health.

1. High-Level Architecture and Data Stores: The architecture leverages an active-passive multi-region setup (Primary Region A handles all transactional writes; Region B serves as hot standby for disaster recovery and regional read traffic). Edge / Virtual Waiting Room: A CDN layer (e.g., Cloudflare) fronts all traffic and integrates an edge virtual waiting room to absorb the initial 500k-user spike. API Gateway: Routes authenticated traffic, performs rate limiting, and validates signed queue admission tokens. Seat Availability Service (Read Path): Reads venue seat status from an in-memory cache (Redis) and serves CDN-cacheable seat map snapshots (updated every 1-2 seconds via SSE or short-polling). Hold and Booking Engine (Write Path): Go/Java microservices executing seat reservation commands against an in-memory transactional cache layer (Redis Cluster) backed by a relational database. Order and Payment Service: Orchestrates order lifecycle, creates payment intents with the payment provider, listens for webhooks, and handles status reconciliation. Data Stores: (a) Redis Cluster: Sharded by Event ID, holds seat state bitmaps and hold leases (TTL: 8 minutes) for microsecond seat locking. (b) Relational Database (Amazon Aurora PostgreSQL Multi-AZ): The authoritative system of record for seats, holds, orders, tickets, and financial ledgers, using ACID transactions to guarantee zero overselling. (c) Kafka / SQS: Asynchronous message bus for order completion events, notifications, ticket PDF generation, and payment reconciliation. 2. Seat Holds, Concurrency, and Expiration: Zero overselling is guaranteed through a two-phase reservation model. In Phase 1 (In-Memory Hold), seat selection requests (up to 6 seats) hit the Hold Engine. A Redis Lua script atomically inspects the bitmask or hash state of the requested seats. If all seats are 'AVAILABLE', it marks them 'HELD', writes a Hold Record containing {user_id, hold_id, timestamp, seat_ids}, and sets a strict 480-second TTL. If any seat is already 'HELD' or 'SOLD', the Lua script aborts cleanly without partial allocation. In Phase 2 (Authoritative Database Hold), an idempotent DB record is created in Aurora within a single transaction: INSERT INTO holds (id, user_id, event_id, seat_ids, expires_at) VALUES (...) and updates seat status to 'HELD'. Expiration Handling: If the user does not finalize purchase within 8 minutes, the Redis hold expires via key expiration events (Redis Keyspace Notifications backed by a durable background reaper scanning Aurora for 'expires_at < NOW() AND status = HELD'). The reaper atomically flips seat status back to 'AVAILABLE' in Aurora and restores the Redis bitmask, instantly releasing seats back to active shoppers. Purchase Finalization: Upon payment confirmation, an ACID transaction in Aurora updates the seats to 'SOLD', marks the order 'PAID', generates immutable ticket records, and updates Redis seat status to 'SOLD'. 3. Traffic Spike, Admission Control, and Fairness: Handling 500,000 concurrent users within 2 minutes requires aggressive admission control so downstream services operate strictly within capacity. Edge Waiting Room: Users arriving before and during the sale opening are directed to a lightweight Edge Waiting Room. Users arriving before T-0 are placed in a pre-sale pool. At T-0, arrival order is randomized within the initial burst to eliminate millisecond bot advantages, after which strict FIFO queuing applies. Rate-Gated Admission: The system calculates admission throughput based on venue size and processing capacity (e.g., admitting 1,000 users every 10 seconds). Admitted users receive a cryptographically signed HMAC token containing {user_id, queue_slot, expires_at} valid for entry to the interactive seat map and checkout pipeline. Seat Map Optimization: Generating 60,000-seat maps dynamically for 500k users would crash databases. Instead, the seat map state is rendered as a compressed binary vector or delta-JSON chunk cached at the CDN edge with a 1-second TTL. Users fetch static venue geometry once, then pull 1-second cache-busted seat status deltas, keeping backend read load under 500 RPS. 4. Payment Integration and Failure Handling: Payments are orchestrated through an asynchronous state machine. Step 1 (Checkout Initiation): User enters payment flow; system creates an internal Order in 'PENDING_PAYMENT' state linked to the Hold ID. An idempotency key (order_id + attempt_count) is generated. Step 2 (External Provider Interaction): The frontend collects payment details using provider client-side SDKs (tokenized cards) and submits the charge intent. Because p99 latency is 3 seconds, API calls run asynchronously with client-side polling on order status rather than synchronous HTTP thread blocking. Timeout and Unclear Results: If the call times out or returns an ambiguous network error, the Order Service does not fail the order immediately. Instead, it places the order in 'PAYMENT_VERIFYING' and initiates a fallback worker that polls the provider's GET /charges/{idempotency_key} endpoint with exponential backoff (e.g., at 2s, 5s, 10s). Hold Extension Buffer: If an order enters 'PAYMENT_VERIFYING' within 60 seconds of the 8-minute hold expiration, the system grants an automatic 2-minute safety extension on the Redis hold to prevent releasing seats while a charge settles. Webhook Ingestion: Webhooks from the payment gateway are validated via HMAC signature and written directly to a durable queue before processing, ensuring idempotent state updates. Reconciliation Worker: An hourly reconciliation job compares payment gateway settlement reports against database records to catch any orphaned charges or unprocessed tickets. 5. Reliability, Failover, and Data Durability: Availability (99.95%) and Zero Data Loss: (a) Database Tier: Aurora PostgreSQL runs across 3 Availability Zones with automated synchronous storage replication and read replicas. If the primary writer fails, automated failover promotes a replica in under 30 seconds. (b) In-Memory Tier: Redis operates in a clustered master-replica topology across AZs. In the rare event of unrecoverable Redis crash, the seat state is rebuilt within 15 seconds by a warmup job executing a fast index scan over the Aurora seats table. (c) Regional Disaster Recovery: Cross-region Aurora storage replication maintains an RPO of under 1 second. Region B is configured in cold/warm standby. If Region A suffers a cloud-provider catastrophic outage, DNS failover redirects traffic to Region B, where services auto-scale and resume operations. (d) Idempotency: All APIs (hold, pay, release) enforce database-level unique constraints and Redis idempotency tokens to prevent duplicate side effects during retries. 6. Key Architecture Trade-Offs: (1) In-Memory Redis Lua Locking vs. Pure Database Row-Level Locking (SELECT FOR UPDATE). Rejected: Direct row-level locks in PostgreSQL for 500k concurrent requests cause massive lock contention, transaction queueing, and connection exhaustion. Chosen: Redis Lua atomic scripts for initial hold arbitration, persisting to Postgres asynchronously/sequentially. Trade-off: Slightly more complex dual-layer state synchronization in exchange for microsecond latency and zero database saturation. (2) Virtual Queue at Edge vs. Massive Auto-Scaling of Backend Services. Rejected: Dynamically scaling Kubernetes pods and databases to handle 500,000 raw RPS for an event with only 60,000 total seats is cost-prohibitive and limited by Aurora write throughput. Chosen: Edge queuing with controlled admission rate. Trade-off: Users wait in a queue, but system stability, operational costs, and user experience remain predictable and robust. (3) CDN-Cached Delta Polling vs. Full Duplex WebSockets for Live Seat Map. Rejected: Maintaining 500,000 persistent bidirectional WebSocket connections places severe stateful connection overhead on application servers. Chosen: Short-cached (1-second) CDN delta polling. Trade-off: 1-second latency in seat state visualization for users, which is entirely tolerable given the transactional Lua hold verification on click. 7. Key Operational Metrics and Alerts: Metrics: (a) Edge Queue Length and Inflow Rate. (b) Hold Creation Rate and Abandonment Rate. (c) Redis Memory Utilization, CPU, and Lua script execution latency (p95, p99). (d) Database Active Connections, Write Latency, and Replication Lag. (e) Payment Gateway Latency (p50, p99), Error Rate (5xx/timeouts), and Webhook Ingestion Delay. (f) Invariant Violation Counter: COUNT(DISTINCT order_id) WHERE seat_id = X AND status = 'SOLD' > 1 (strictly 0). Critical Alerts: (1) Invariant Breach: Any detection of oversold seat (P0, instant page). (2) Payment Timeout Spike: Payment timeout rate > 5% over 1 minute (P1). (3) Redis Memory / CPU: CPU > 75% or eviction count > 0 (P1). (4) Expired Hold Reaper Lag: Reaper queue delay > 30 seconds (P2).