Design a Scalable Real-Time Notification System

Design: Scalable Real-Time Notification System 1. OVERALL ARCHITECTURE The system is composed of several distinct layers that work together to ingest events, route them, and push them to connected clients with minimal latency. Client Layer: Mobile and web clients maintain persistent WebSocket connections to a fleet of Connection Gateway servers. Each client authenticates on connect and registers its user ID with the gateway. API Gateway / Load Balancer: A Layer-7 load balancer (e.g., AWS ALB or NGINX) sits in front of the Connection Gateways. It routes new WebSocket upgrade requests using consistent hashing on the user ID so that reconnects tend to land on the same gateway node, reducing state churn. It also exposes a REST endpoint for internal services to publish events. Event Publishing Service: Internal platform services (like service, comment service, friend service) publish events to a central message broker. They call a thin Publishing API that validates the payload, enriches it with metadata (timestamp, notification ID), and writes it to the broker. Message Broker (Kafka): Events are written to Kafka topics partitioned by user ID. This ensures ordered delivery per user and allows horizontal scaling of consumers. Kafka's durable log also provides the replay capability needed for at-least-once delivery guarantees. Notification Fanout Service: A pool of stateless consumer workers reads from Kafka. For each event, the worker looks up the target user's subscription preferences in a fast cache (Redis), determines which users should receive the notification, and then routes the message to the correct Connection Gateway. For high-fanout events (e.g., a celebrity post), a separate async fanout job is triggered to avoid blocking the hot path. Connection Gateway (WebSocket Servers): These are stateful servers that maintain the open WebSocket connections. Each gateway holds an in-memory map of user ID to connection handle. When a routed notification arrives (via an internal pub/sub channel like Redis Pub/Sub or a direct gRPC call), the gateway pushes it down the appropriate WebSocket connection. If the user is not connected, the gateway discards the push and relies on the persistence layer for later delivery. Presence & Routing Service: A Redis cluster stores a mapping of user ID to gateway node ID with a short TTL, refreshed by heartbeats. The Fanout Service queries this to know which gateway to route a notification to. If no entry exists, the user is offline. Notification Storage (Cassandra): All generated notifications are written to Cassandra, keyed by user ID and sorted by timestamp. This serves two purposes: it powers the notification inbox UI (users can scroll back through past notifications) and it enables at-least-once delivery — when a user comes online, the client fetches unread notifications from this store. Delivery Acknowledgment: Clients send an ACK message over the WebSocket after receiving a notification. The gateway writes this ACK to Kafka, and a consumer marks the notification as delivered in Cassandra. Unacknowledged notifications older than a threshold are re-queued for delivery. 2. TECHNOLOGY CHOICES AND REASONING WebSockets over Long Polling or SSE: WebSockets provide full-duplex, low-overhead persistent connections. Long polling wastes server resources with repeated HTTP handshakes and adds latency. Server-Sent Events (SSE) are unidirectional and less suitable for the ACK flow. At 1 million concurrent connections, WebSockets are the most resource-efficient choice. Each connection consumes roughly 10–50 KB of memory, making 1 million connections feasible across a moderately sized gateway fleet. Kafka over RabbitMQ: Kafka is chosen for its high throughput (millions of messages per second), durable log storage, consumer group semantics, and the ability to replay messages. RabbitMQ is a good broker for task queues but its message model is less suited to the fan-out and replay patterns needed here. Kafka's partitioning by user ID also naturally parallelizes consumption. At 10,000 notifications per second, Kafka handles this with significant headroom. Redis for Presence and Pub/Sub: Redis provides sub-millisecond reads for the presence lookup (user ID → gateway node). Redis Pub/Sub or Redis Streams can be used for the internal channel between the Fanout Service and the Connection Gateways, adding minimal latency to the delivery path. Cassandra over MySQL/PostgreSQL: Notification history is a write-heavy, time-series workload with high cardinality (one partition per user). Cassandra's wide-column model, tunable consistency, and linear horizontal scalability make it ideal. A relational database would require complex sharding and struggle with the write throughput. Cassandra's eventual consistency is acceptable here since notification history is not a transactional record. Stateless Fanout Workers: Keeping the fanout workers stateless allows them to scale horizontally by simply adding more Kafka consumer instances within the consumer group. 3. HOW THE DESIGN MEETS EACH REQUIREMENT Scalability (1M concurrent users, 10K notifications/second): The Connection Gateways are horizontally scalable. A single modern server can hold 50,000–100,000 WebSocket connections, so 10–20 gateway nodes handle 1 million users. The load balancer distributes new connections. Kafka partitions scale the fanout workers. Cassandra scales writes linearly with nodes. Redis Cluster shards the presence data. No single component is a bottleneck. Latency (P99 < 200ms): The critical path is: event published → Kafka write (~5ms) → Fanout worker consumes and looks up presence in Redis (~5ms) → routes to gateway via Redis Pub/Sub or gRPC (~5ms) → gateway pushes over WebSocket (~10ms network). Total is well under 50ms in the median case. The 200ms P99 budget accommodates Kafka consumer lag under peak load and network jitter. Keeping the fanout worker logic simple and the Redis lookups cached ensures the hot path stays fast. Reliability (at-least-once delivery): Notifications are persisted to Cassandra before or concurrently with the push attempt. If the WebSocket push fails or the client does not ACK, the notification remains in the unread state in Cassandra. On reconnect, the client fetches unread notifications. Kafka's consumer offset commit is done only after the fanout worker has successfully routed the message, ensuring no event is silently dropped. This provides at-least-once semantics end to end. Availability (99.95% uptime): All components are deployed in multiple availability zones. The load balancer, Kafka brokers, Redis Cluster nodes, Cassandra nodes, and fanout workers all run with N+1 or N+2 redundancy. Gateway node failures cause clients to reconnect (WebSocket reconnect logic with exponential backoff) and land on a healthy node within seconds. Kafka replication factor of 3 ensures broker failures do not cause data loss. Cassandra's replication factor of 3 with quorum reads/writes tolerates node failures. This architecture comfortably achieves 99.95% uptime. 4. TRADE-OFFS Complexity vs. Simplicity: This design has many moving parts — Kafka, Redis, Cassandra, WebSocket gateways, fanout workers, presence service. This is significantly more complex to operate than a simple polling system or a single-broker setup. The trade-off is justified by the scale requirements, but it demands a mature DevOps practice, good observability (distributed tracing, metrics per component), and on-call expertise. At-Least-Once vs. Exactly-Once: Exactly-once delivery would require distributed transactions across Kafka, Cassandra, and the gateway, adding significant latency and complexity. At-least-once is chosen instead, meaning a user might occasionally see a duplicate notification. This is handled on the client side by deduplicating on notification ID. For a social media notification (a like or comment), a duplicate is a minor UX annoyance, not a critical failure — an acceptable trade-off. Stateful Gateways: The WebSocket gateways are stateful (they hold live connections). This makes rolling deployments and failure recovery slightly more complex than a fully stateless service. Clients must implement reconnect logic. The trade-off is that stateful gateways are necessary for real-time push; there is no way to push to a client without a persistent connection. Fanout at Read vs. Write Time: This design fans out at write time (the fanout worker pushes to each recipient's gateway immediately). For users with millions of followers, this creates a hot path problem (a celebrity posting triggers millions of fanout operations). The mitigation is to use a separate high-fanout job queue for accounts above a follower threshold, processing them asynchronously. The trade-off is that high-follower users may experience slightly higher notification latency, which is generally acceptable. Redis Pub/Sub Reliability: Redis Pub/Sub does not persist messages. If a gateway is briefly disconnected from Redis when a message is published, it misses that message. This is mitigated by the Cassandra persistence layer — the client will fetch the notification on reconnect — but it means the real-time push is best-effort, with the persistent store as the safety net. Using Redis Streams instead of Pub/Sub would add persistence at the cost of slightly more complexity. Cost: Running Kafka, Redis Cluster, Cassandra, and a large WebSocket gateway fleet is expensive. A smaller platform might use a managed service like AWS SNS/SQS or Firebase Cloud Messaging to reduce operational burden. At the scale described (1M concurrent users), the control and performance of a custom stack justifies the cost.

High-level summary I propose a horizontally scalable, low-latency real-time notification platform built around: (1) edge/gateway infrastructure to accept events and client connections, (2) a durable pub/sub backbone for routing events to interested subscribers, (3) connection managers / WebSocket workers that maintain live client connections and perform fan-out, and (4) a fast presence store and durable notification store for reliability and offline delivery. The system uses partitioning and replication to meet throughput and availability goals, with end-to-end instrumentation, backpressure, and idempotence to enforce at-least-once delivery and low latency. 1) Overall architecture (components and request flow) Core components - API Gateway / Ingress: Accepts event publishes (from internal services) and client connection upgrades. Exposes REST/gRPC endpoints for event publishers and handles authentication and rate limiting. Deployed behind global LBs and per-region endpoints. - Connection Layer (WebSocket / GRPC streams): A fleet of stateless connection servers (WebSocket/HTTP2 or gRPC) that maintain persistent connections to clients. They perform lightweight routing and acknowledgment handling and forward subscription changes to the presence store. - Presence & Routing Store: A low-latency key-value store (Redis cluster) that tracks which connection server(s) currently host each online user and which topics they subscribe to. Used to route notifications to the correct connection worker. - Pub/Sub Backbone: A durable, partitioned message bus (Kafka or Apache Pulsar) used for distributing events from publishers to notification workers. Topics are partitioned by logical keys (user id, topic id) to ensure ordered processing per key and scalable throughput. - Notification Service / Worker Pool: Consumers of the pub/sub topic that perform enrichment, filtering, and delivery routing. Workers look up the presence store to find target connection servers and push delivery tasks into fast delivery paths. - Delivery Layer / Fan-out Engine: The connection servers receive delivery requests (directly or via fast RPC) and push notifications over the persistent connection to the client. They handle per-connection flow control, batching, and ACKs from clients. - Durable Notification Store (NoSQL): A write-optimized, replicated store (e.g., Cassandra / DynamoDB) to persist notifications for offline users, retries, and audit. Stores notification payloads, delivery attempts, timestamps, TTLs. - Deduplication & Idempotence Store: Small key-value store (Redis or RocksDB) to record recent message IDs for deduplication when at-least-once semantics cause duplicates. - Monitoring & Control Plane: Metrics, tracing, SLO/alerting, circuit breakers, and throttling. Typical flows - Publish flow (event producer -> user): 1) Publisher posts event to API Gateway (REST/gRPC). 2) Gateway writes message to Pub/Sub topic (partitioned by target topic/user id). 3) Notification workers consume events, enrich and resolve subscription lists (or query subscription DB), consult presence store to find online recipients, and for each online connection push to the appropriate connection server(s). 4) Connection servers push over WebSocket/gRPC stream to client and await lightweight ACK. 5) If user offline (presence miss), write to durable Notification Store for offline retrieval. - Subscribe flow: Clients send subscribe/unsubscribe messages over their persistent connection. Connection server updates presence store atomically so the worker routing sees updated subscriptions quickly. - Recovery & replay: Durable pub/sub and Notification Store allow replay of missed events; connection servers re-register presence on reconnect. 2) Technology choices and rationale - Client connection: WebSockets or HTTP/2 (gRPC) streams - Choice: WebSockets for broad client compatibility (mobile/web). Consider HTTP/2 gRPC streams for internal/mobile apps supporting it. Both keep persistent TCP/TLS connections to meet <200ms latency. - Reason: Polling is too slow and inefficient. Long polling increases latency and resource usage. WebSockets provide push, low RTT, and ability to deliver binary messages with small overhead. - Pub/Sub backbone: Apache Kafka or Apache Pulsar - Choice: Kafka (managed like Confluent Cloud or self-hosted) or Pulsar if multi-tenancy and geo-replication needs dominate. - Reason: Both provide partitioned, durable, high-throughput messaging. Kafka has mature tooling, strong throughput, predictable latency, and exactly-once semantics support in producers. Partitioning allows scaling to 10k msg/s easily. Pulsar offers built-in geo-replication if multi-region is required. - Presence store & routing: Redis Cluster (in-memory) with replication - Reason: Sub-1ms reads/writes to map user -> connection servers, subscriptions, and connection metadata. Redis supports clustering, persistence (AOF/RDB), and fast lookups needed to route events within the 200ms budget. - Persistent notification store: Cassandra / DynamoDB (NoSQL) - Choice: Cassandra or DynamoDB-like key-value store. - Reason: High write throughput, linear scalability, and configurable TTLs, ideal for storing many writes per second and serving offline reads. SQL systems struggle at this scale of writes and horizontal sharding complexity. - Connection servers & worker communication: gRPC/internal RPC + protobuf - Reason: Low overhead binary messages, backpressure support, strong typing, and high perf. - Load balancing & routing: L4 (TCP) load balancers for WebSockets + L7 LB for REST APIs. Use sticky routing via consistent hashing or session affinity to route reconnections to same region. - Client payload format: Compact binary (protobuf/flatbuffers) for smaller payloads and faster serialization. 3) How the design meets non-functional requirements - Scalability (1M concurrent users, 10k notif/s peak) - Stateless connection servers: Horizontal scale-out; each server handles N concurrent WebSocket connections. With modern machines (e.g., 100k sockets per machine with proper tuning), a few dozen/hundreds of servers handle 1M concurrent sockets. Autoscaling group + container orchestration manages capacity. - Partitioned pub/sub (Kafka): Scale producers and consumers by adding partitions/workers. Partition by target user ID or topic ensures even load distribution. 10k msg/s is modest for Kafka clusters sized appropriately (dozens of brokers). - Sharded presence store: Redis Cluster shards user space so lookups remain O(1) and scale. - Partitioned durable store: Cassandra scales linearly by adding nodes for writes and reads. - Latency (99% within 200ms) - Persistent connections eliminate TCP handshake overhead. Once established, push from event publish to client is: API Gateway -> Kafka append (sub-ms-10s ms depending on tuning) -> worker consume -> presence lookup (sub-ms) -> RPC to connection server -> push over WebSocket (sub-ms to a few ms). With careful placement (deploy workers and connection servers in same region/az), network latency remains small. Use batching tuned to near-instant delivery (avoid batching delays) and tune Kafka consumer configuration (low linger.ms) to keep end-to-end latency low. - Edge/regional deployment: Place connection servers near clients (region-level) so last-mile latency is minimized. - Use binary serialization and small payloads to reduce serialization/network time. - Reliability (at-least-once delivery) - Durable writes: Publishers write events to Kafka (durable), workers track offsets; messages remain durable until acknowledged by consumer processing. - At-least-once: If delivery fails (connection server crash, network), the worker can re-enqueue or retry because the pub/sub retains messages and the worker commits offsets only after successful enqueue/confirmation. Clients ACK receipts, but ACKs only confirm client acceptance; server-side persistence allows retries. - Deduplication: Since at-least-once can produce duplicates, include unique message IDs and client-side dedupe or server-side dedupe using recent-ID caches to suppress duplicates. - Durable Notification Store: When user offline or when persistent confirmation is required, write the notification to Cassandra before acknowledging the event so it can be delivered later. - Availability (99.95% uptime) - Multi-AZ and multi-region deployment: Replicate critical components. Use Kafka with replication factor >2, Redis with master-replica and failover, Cassandra with multiple replicas and tunable consistency. - Stateless frontends & autoscaling: If any connection server fails, clients reconnect to next available server. Use health checks and fast failover. - Graceful degradation: If delivery overload occurs, accept events and persist them for later delivery, rather than dropping. Throttling and backpressure protect the system. - Observability & automation: Auto-restart, circuit breakers, and runbooks for operator intervention. SLOs and alerting tuned to maintain 99.95% availability. 4) Operational details and optimizations - Partitioning keys & routing: Partition pub/sub by user id for user-targeted messages and by topic id for topic broadcasts. For fanouts to many users (e.g., a post liked by thousands), perform hierarchical fan-out: 1) determine recipients; 2) group recipients by connection server; 3) send grouped delivery message to each connection server to avoid NRPC calls. - Fan-out strategies - Fan-out on write (push): Preferred for real-time online delivery — workers push only to online connections found via presence store. - Fan-out on read (pull): Use for large offline fanout or for infrequently-online users; store notification and let clients fetch when online. - Batching and coalescing: For high-volume similar events, coalesce multiple events into one compact notification when safe (e.g., “3 people liked your post”) to reduce load and improve UX. - Backpressure and smoothing: If connection servers or client cannot accept messages, apply backpressure: slow down consumers, buffer to durable store, and retry. Implement per-client rate limits. - Client ACK model: Use a lightweight ACK for successful receipt. If ACK not received within timeout, retry delivery. Maintain delivery attempt counters and send to dead-letter store after N attempts. - Security and privacy: End-to-end auth at gateway, verify publisher entitlement to publish; encrypt transport (TLS), and sanitize payloads. 5) Trade-offs and discussion - Complexity vs Simplicity - Trade-off: A Kafka-based, multi-component system is more complex operationally than a simple push server but necessary for scale and reliability. Operational cost and engineering complexity increase (managing Kafka, Redis cluster, Cassandra), but provide durability, replay, and fanout control. - At-least-once vs Exactly-once - Choice: At-least-once delivery with deduplication is chosen. Exactly-once end-to-end is very expensive (coordination across services and clients) and often unnecessary for notifications where duplicates are tolerable. Deduplication caches and client-side idempotence provide practical mitigation. - WebSockets vs HTTP/2 gRPC - Trade-off: WebSockets maximize compatibility but require more LB and connection tuning. gRPC offers better flow-control and type safety for clients that support it. Supporting both adds complexity but gives best coverage. - Real-time push vs storing-first - Push-first is prioritized for online users to meet latency. However, storing-first (persisting to DB first) adds durability at cost of additional write latency. We strike a balance: write to Kafka (durable) quickly, optionally persist to Cassandra depending on reliability policy. - Redis presence eventual consistency - Presence may be slightly stale in edge cases; a small window of false negatives/positives could cause a single missed notification which will be recovered from the durable notification store. This design favors low-latency presence lookups over strictly-synchronous global presence values. 6) Capacity planning & numbers (back-of-envelope) - 1M concurrent users: If each connection server handles ~10k concurrent WebSockets, need ~100 connection servers. Add safety factor (2x) for peaks -> ~200 instances across regions. - 10k notifications/s: Kafka cluster with 10–50 partitions is sufficient depending on message size and replication; target broker count around 5–10 with replication factor 3. Consumer groups scale horizontally. - Storage: At 10k notif/s, ~864M notifications per day (if every message persisted) — use TTLs and aggregation to reduce storage. Cassandra nodes sized accordingly. 7) Failure modes and mitigations - Connection server crash: Clients reconnect to other servers; presence store updated on reconnect. Buffered messages in worker are retried. Use sticky reconnection strategies to reduce state churn. - Kafka broker failure: Partition replication ensures no data loss when replication factor >=3 and ISR configured. Monitor under-replicated partitions. - Redis failover: Use Redis Cluster with replicas and automated failover (Sentinel or managed services). Rebuild presence from connection servers on large failover. - Large fanouts (viral events): Use hierarchical fanout and batching; offload to background workers and persist notifications for later delivery if real-time delivery would breach latency. Conclusion This design uses persistent connections (WebSockets/gRPC), a durable partitioned pub/sub (Kafka/Pulsar), a low-latency presence store (Redis), and a scalable NoSQL store (Cassandra/DynamoDB) to meet the 1M concurrent connections, 10k notif/s, <200ms 99% latency, at-least-once reliability, and 99.95% availability goals. The main trade-offs are increased operational complexity and handling duplicates. With careful partitioning, replication, monitoring, and per-component tuning (consumer configs, batching, backpressure), this architecture will scale and provide the low-latency, reliable real-time notifications the platform needs.