Design a Real-Time E-commerce Notification System

Overview I propose a horizontally scalable, streaming-first notification platform built from decoupled microservices and managed cloud services. Core ideas: use a durable event log for ingestion and replay (Kafka/Kinesis/Pub/Sub), stream processors to enrich and personalize events, a delivery pipeline that persists delivery state and retries for at-least-once guarantees, and device-specific connectors for push (APNs/FCM) and real-time web (WebSocket). The design meets 100k notifications/min (≈1.7k/sec) peaks, targets sub-5s delivery for 99% of messages, and supports personalization and reliable delivery. High-level architecture (components and interactions) 1) Event Producers - Sources: Order service (order updates), Pricing service (price changes), Marketing/CRM (flash sales). Each service emits lightweight events to the ingestion layer whenever a relevant change occurs. Events include event_id, event_type, payload, timestamp, and metadata (user_ids or product_ids). 2) Ingestion / Durable Event Log - Managed partitioned log: Apache Kafka (self-managed or Confluent Cloud) or cloud equivalents (AWS Kinesis Data Streams, GCP Pub/Sub). Producers publish events to topic(s) organized by event type and partition key (user_id or product_id) to preserve ordering where required (e.g., order updates per order). - Why durable log: provides replayability, retention for retries, and backpressure smoothing. 3) Stream Processing / Enrichment Layer - Stateless/Stateful stream processors (Apache Flink, Kafka Streams, or managed Dataflow) subscribe to event topics to: validate events, enrich with user profile and preferences, join with product/segment data, and decide notification eligibility and priority (e.g., critical order update vs marketing). - Output: normalized Notification Tasks (task_id, user_id(s), payload, type, priority, ttl, dedup_key) published to a Notification Task topic. 4) Personalization & Segmentation - Personalization rules live in a service combining: feature store / profile DB (DynamoDB/Cassandra/Postgres + Redis cache for hot reads), and a rule engine or ML model. Stream processors call this service or use local cached lookups to determine targeted recipients and content variants. - For broad segmentation events (flash sale to segment), use precomputed segments stored in a fast store (Redis, Druid, or BigQuery/ElastiCache lookup) to expand to user lists or to apply filter logic within streaming jobs. 5) Delivery Orchestration / Fan-out - A Delivery Orchestrator service subscribes to Notification Task topic, evaluates device registrations, throttling rules, and fan-out strategy. For single-user notifications (order update) it creates a delivery job per device; for segment-based broadcast it fans out into many delivery jobs via a partitioned queue. - Delivery jobs are placed into persistent per-shard delivery queues (Kafka topics, Redis Streams, or SQS with FIFO for ordering where needed). Jobs include retry counters and idempotency/dedup keys. 6) Delivery Workers / Connectors - Stateless worker fleet autoscaled by queue lag. Each worker pulls jobs, attempts delivery via the connector appropriate for the device channel: - Mobile push: FCM (Android) and APNs (iOS) using device tokens stored in Device Registry. - Web/Browser: Web Push (VAPID) or persistent WebSocket connections (managed via a connection service like AWS API Gateway WebSocket or self-managed socket clusters behind ELB). - Fallback channels: Email (SES/SendGrid) or SMS (Twilio) for critical undelivered notifications. - Workers persist delivery attempts (success/fail) to a Delivery Status store and emit completion or retry events to the log for monitoring and further retries. 7) Device Registry & User Preferences - Durable store of user_id -> devices (token, platform, last_seen, preferences, opt-in flags). Use DynamoDB/Cassandra for high write throughput; cache active devices in Redis for low-latency lookups. 8) Delivery State & Replayability - All notification tasks and delivery attempts logged in durable stores (Kafka + archival to S3) and a Delivery Status DB. This enables at-least-once delivery, auditing, and reconciliation. Unacked/failed deliveries are retried by a retry orchestrator with exponential backoff. 9) Monitoring, Observability, and SLA Enforcement - Metrics: ingestion rate, processing latency, queue lag, delivery success rate. Traces for path-level latency (OpenTelemetry), and alerts for SLA breaches. Dashboards to monitor 99p latency and per-channel failure rates. Key design choices and justifications - Durable log (Kafka/Kinesis/PubSub): provides high throughput and replayability which is essential for at-least-once semantics and debugging. Partitioning by user_id/product_id preserves per-entity ordering (critical for order updates). Managed cloud streaming reduces operational overhead. - Stream processing (Flink/Kafka Streams/Dataflow): enables sub-second enrichment and segmentation close to ingestion. Stateful streaming supports windowed joins (e.g., match price drop events to wishlists) with low latency. - Device Registry in NoSQL + cache: DynamoDB/Cassandra scales horizontally for tens of millions of users; Redis handles hot-path lookups for low-latency decisions. - Delivery queues and autoscaled workers: decouples heavy fan-out from upstream processing, enabling graceful scaling during flash sales while controlling downstream push provider rate limits. - Push connectors (APNs/FCM) + WebSockets: push services minimize client polling and achieve low latency. WebSockets are used for real-time in-app/web delivery; if WebSocket unavailable, fall back to push or pull. - At-least-once, idempotency and deduplication: store task-level dedup_key and make delivery idempotent on the client or use SDK acknowledgements where possible. On server-side, dedupe by task_id/dedup_key before creating user-visible notifications. Meeting the requirements - High Throughput: Partitioned log and autoscaling workers support horizontal scaling; Kafka/Kinesis can handle millions of events/sec with multiple partitions. 100k/min is modest for such systems; the architecture can scale to much higher volumes by adding partitions and workers. - Low Latency: Streaming enrichment and direct push/WebSocket connectors are low-latency paths. Targeting <5s 99p: keep processing pipeline under 1–2s (streaming jobs), delivery queue lag low via autoscaling workers, and use device caches to avoid DB lookups in the hot path. - Reliability: Durable event log + persisted delivery states + retry orchestrator ensures at-least-once delivery. For critical notifications (order updates), enable stronger guarantees: synchronous acknowledgement from downstream services and storing a confirmed delivery receipt (e.g., device ack or fallback channel confirmation). Use exponential backoff and escalation to alternate channels. - Scalability: All stateful pieces use horizontally scalable stores (Kafka, DynamoDB/Cassandra, Redis clusters). Workers and streamers are stateless containers that autoscale. Use partitioning and sharding for growth. - Personalization: Real-time joins in stream processors plus cached profile store enable per-user personalization. Precomputed segments accelerate large fan-outs (flash sales) by avoiding per-user evaluate on the fly. Trade-offs (Consistency, Availability, Cost) - Consistency vs Availability: We favor availability and eventual consistency for marketing notifications (acceptable if a promo arrives slightly out of order). For order-critical events, we use stronger ordering and persistence (partitioning and synchronous persistence) to ensure correct ordering and reliable delivery. This hybrid approach balances user experience and system resilience. - At-least-once vs Exactly-once: Achieving exactly-once across the whole pipeline adds complexity and cost (transactional Kafka, two-phase commit, or end-to-end idempotency). We choose at-least-once with idempotent handlers and dedup keys to avoid duplicate-visible notifications while keeping system simpler and more scalable. - Managed services vs self-hosted: Managed streaming (Kinesis/PubSub) and push infrastructure reduce operational burden and increase availability but cost more. For speed to market and reliability at scale, managed services are recommended. If cost becomes dominant, consider self-hosted Kafka with strong automation. Operational considerations - Rate limiting / throttling: Per-user and per-provider quotas to avoid overload and provider rate-limit rejections. - Backfill and replay: Use durable log to replay missed notifications (e.g., after outage) and to rebuild delivery status. - Testing: Chaos testing for partitions and provider failures, load testing to validate 99p latency under peak. - Security & privacy: Encrypt tokens at rest, rotate push credentials, honor opt-outs, and PII controls. SLA and KPIs to monitor - Throughput (events/min), stream processing latency (p50/p90/p99), end-to-end delivery latency (p50/p90/p99), delivery success rate per channel, retry rate, queue lag, and consumer group lag. Summary This design uses a durable, partitioned event log plus stream processing for enrichment/personalization, a persistent delivery pipeline with autoscaling workers for fan-out, and channel-specific connectors (APNs/FCM/WebSocket) for fast delivery. It prioritizes availability and replayability to meet at-least-once delivery, uses caching and partitioning to meet low latency and high throughput, and relies on managed cloud components to reduce operational burden while allowing horizontal scaling to support future growth.

Designing a Real-Time E-commerce Notification System Overview The proposed system is an event-driven, microservices-based architecture that ingests events from multiple upstream services, processes and routes them through a message broker, and delivers notifications to 50 million users across mobile and web channels with at-least-once delivery guarantees, sub-5-second latency at the 99th percentile, and the ability to sustain 100,000 notifications per minute during peak events. The design assumes a cloud-native deployment on AWS, though the principles translate directly to GCP or Azure equivalents. High-Level Architecture The architecture consists of six logical layers: Event Producers, Event Ingestion, Processing and Routing, User Preference and Segmentation, Delivery, and Observability. 1. Event Producers Multiple upstream services generate events that trigger notifications. These include the Order Service (emitting order-placed, shipped, out-for-delivery, delivered events), the Pricing Service (emitting price-drop events when a watched item's price falls below a threshold), and the Campaign Service (emitting flash-sale-start events created by the marketing team). Each producer publishes a structured event envelope containing an event type, a timestamp, a payload, and an idempotency key to a central ingestion layer. Producers use an asynchronous, fire-and-forget publish call so their own latency is unaffected. 2. Event Ingestion Layer — Apache Kafka All events flow into Apache Kafka (Amazon MSK in AWS). Kafka is chosen for several reasons. First, it provides durable, append-only logs with configurable replication (replication factor of 3, min.insync.replicas of 2), which means an acknowledged write survives the loss of any single broker, directly supporting the reliability requirement. Second, Kafka's partitioned topic model allows horizontal scaling: we partition the order-events topic by user ID so that all events for a given user are processed in order, while the overall topic can be spread across dozens of partitions to absorb peak throughput. At 100,000 notifications per minute, the average rate is roughly 1,667 events per second, well within a modest Kafka cluster's capacity, but partitioning lets us scale to 10x or more without architectural change. Third, Kafka decouples producers from consumers, so a temporary slowdown in the delivery layer does not back-pressure the order service. Topic design: separate topics for order-events, price-events, and campaign-events. This allows independent consumer groups with different scaling and retry policies. Trade-off considered: We evaluated Amazon SQS plus SNS as a simpler managed alternative. SQS would satisfy at-least-once delivery and is operationally lighter, but it lacks Kafka's ordering guarantees per partition and its replay capability, which is valuable for reprocessing after a bug. The added operational cost of Kafka is justified by the ordering and replay benefits at our scale. 3. Processing and Routing Layer — Notification Service A horizontally scaled microservice, the Notification Service, consumes from Kafka topics. It performs several tasks. First, it resolves the target audience. For order events, the target is a single user (extracted from the payload). For price-drop events, it queries the Wishlist Service or a precomputed materialized view to find all users watching that item. For flash-sale events, it queries the Segmentation Service to resolve a user segment. Second, it enriches the notification by fetching the user's display name, the product image URL, and the localized message template from a Template Service backed by a Redis cache over a PostgreSQL store. Third, it applies user preferences. Each user has a preference document (stored in DynamoDB for low-latency key-value access) specifying opted-in channels (push, email, SMS, in-app), quiet hours, and category interests. The service filters and respects these preferences, which directly supports the personalization requirement. Fourth, it fans out delivery tasks. For each resolved user-channel pair, the service produces a delivery message onto a per-channel Kafka topic (push-delivery, email-delivery, sms-delivery, in-app-delivery). This fan-out step is critical: it converts one logical event into potentially millions of individual delivery tasks (for a flash sale targeting all users), and Kafka absorbs this burst. Scaling: The Notification Service runs as a Kubernetes Deployment (Amazon EKS) with a Horizontal Pod Autoscaler keyed on Kafka consumer lag. During a flash sale, consumer lag spikes, and additional pods spin up within seconds. Idempotency: Every delivery message carries the original event's idempotency key combined with the user ID. Downstream delivery workers use this composite key to deduplicate, ensuring that even though Kafka provides at-least-once semantics, users do not receive duplicate notifications. 4. User Preference and Segmentation User preferences are stored in Amazon DynamoDB, partitioned by user ID. DynamoDB is chosen for its single-digit-millisecond read latency and seamless horizontal scaling, which is important because every single notification requires a preference lookup. A DAX (DynamoDB Accelerator) cache sits in front for hot keys. For segmentation (targeting users interested in a category, or users in a geographic region), we maintain precomputed segment membership lists. A nightly batch job (Apache Spark on EMR) and a real-time stream processor (Kafka Streams or Flink) keep these lists updated in a separate DynamoDB table keyed by segment ID, with the value being a list of user IDs stored in S3 for very large segments. When the Campaign Service creates a flash sale targeting the electronics segment, the Notification Service reads the segment membership and iterates through it, producing delivery tasks in batches. Trade-off: Precomputing segments trades storage and staleness (a user who changed preferences an hour ago might still be in the old segment) for query speed. Real-time segment evaluation at delivery time would be more accurate but would require scanning millions of user records under time pressure, violating the latency requirement. The hybrid approach (nightly batch plus real-time stream updates) keeps segments fresh within minutes. 5. Delivery Layer Separate consumer groups handle each channel. Push Notifications (Mobile): Workers consume from the push-delivery topic and call Firebase Cloud Messaging (FCM) for Android and Apple Push Notification Service (APNs) for iOS. We use FCM as the unified gateway for both platforms where possible. Workers batch requests to FCM's HTTP v1 API (up to 500 messages per batch call) to maximize throughput. Failed deliveries (invalid tokens, rate limits) are retried with exponential backoff. Permanently failed tokens (unregistered devices) trigger an asynchronous event to purge the token from the User Device Registry. Web Push: Workers send Web Push Protocol messages using the VAPID standard. The user's push subscription (endpoint URL and keys) is stored in the User Device Registry (DynamoDB). This channel reuses the same push-delivery topic with a channel sub-type field. In-App Notifications: For users who are currently online, we maintain persistent WebSocket connections via an API Gateway WebSocket API (AWS API Gateway WebSocket or a self-managed service using Socket.IO on EKS). A Connection Registry (Redis Cluster) maps user IDs to WebSocket connection IDs. The in-app delivery worker looks up the connection, and if the user is connected, pushes the notification in real time. If not connected, the notification is written to an In-App Inbox (DynamoDB table partitioned by user ID, sorted by timestamp) and delivered when the user next opens the app. This dual-write ensures no notification is lost even if the user is offline. Email and SMS: Lower-priority channels. Workers consume from email-delivery and sms-delivery topics and call Amazon SES and Amazon SNS (SMS) respectively. These channels have higher acceptable latency (30 seconds to minutes), so they can be scaled more conservatively. Delivery Guarantees: At-least-once delivery is achieved end-to-end. Kafka consumers commit offsets only after the downstream delivery API acknowledges receipt (or the message is persisted to the in-app inbox). If a worker crashes before committing, the message is redelivered. The idempotency key at each stage prevents user-visible duplicates. 6. Notification Status Tracking A lightweight Status Service records the lifecycle of each notification: created, sent, delivered, read. Delivery receipts from FCM/APNs and read receipts from the client app are ingested via a Kafka topic and written to a time-series store (Amazon Timestream or ClickHouse) for analytics, and to DynamoDB for per-user status queries (so the app can show read/unread badges). This data also feeds retry logic: if a push notification is not acknowledged as delivered within 30 seconds, the system can fall back to email. 7. Observability Prometheus and Grafana monitor Kafka consumer lag, delivery latency percentiles, error rates per channel, and throughput. Alerts fire if p99 delivery latency exceeds 4 seconds (giving a 1-second buffer before the 5-second SLA). Distributed tracing (OpenTelemetry with Jaeger) tracks an event from producer through to delivery, enabling rapid debugging. Structured logs are shipped to Amazon OpenSearch for search and audit. Addressing the Requirements High Throughput: Kafka's partitioned topics and horizontally scaled consumer groups handle 100K notifications per minute comfortably. The fan-out for large campaigns is absorbed by Kafka's buffering, and delivery workers auto-scale based on lag. Low Latency: The critical path for a single-user notification (e.g., order shipped) is: producer publishes to Kafka (less than 50ms), Notification Service consumes and enriches (less than 100ms including DynamoDB and Redis lookups), delivery worker sends to FCM/APNs (less than 500ms typical). Total well under 5 seconds. For large fan-out campaigns, the system parallelizes across many delivery workers; Kafka partitioning ensures no single worker is a bottleneck. Reliability: Kafka replication, consumer offset management, and idempotent delivery ensure at-least-once semantics. A dead-letter queue (DLQ) topic captures messages that fail after a configurable number of retries, and an ops team is alerted to investigate. Critical order notifications are flagged with a priority field and routed to a dedicated high-priority partition with its own consumer group, ensuring they are never starved by a flood of promotional notifications. Scalability: Every component scales horizontally. Kafka partitions can be increased. Kubernetes pods auto-scale. DynamoDB scales on demand. There are no single points of failure or single-threaded bottlenecks. Personalization: User preferences in DynamoDB control channel selection and quiet hours. Precomputed segments enable targeted campaigns. The Template Service supports per-locale, per-category message templates, and A/B test variants can be selected at enrichment time. Key Trade-offs and Justifications Consistency vs. Availability: The system favors availability (AP in CAP terms). If the preference store is momentarily unreachable, the Notification Service uses a cached or default preference rather than blocking. This means a user might occasionally receive a notification on a channel they just opted out of, but no notification is lost or delayed. For an e-commerce notification system, availability and low latency are more valuable than strict consistency of preferences. At-Least-Once vs. Exactly-Once: True exactly-once delivery across external systems (FCM, APNs) is impractical. Instead, we guarantee at-least-once and use idempotency keys to suppress duplicates at the application level. This is simpler, cheaper, and sufficient. Managed Services vs. Self-Hosted: We use managed Kafka (MSK), managed Kubernetes (EKS), and managed databases (DynamoDB) to reduce operational burden. The trade-off is higher per-unit cost compared to self-hosted, but the engineering time saved and the reliability of managed services justify this at our scale. Push vs. Pull for In-App: WebSockets (push) provide the lowest latency for online users but require maintaining persistent connections and a connection registry. Polling would be simpler but would add seconds of latency and generate unnecessary load. The hybrid approach (WebSocket when online, inbox when offline) balances latency and resource usage. Cost Considerations: The largest cost drivers are Kafka throughput (MSK pricing), DynamoDB read capacity for preference lookups, and FCM/APNs API calls (free for FCM, but infrastructure to call them at scale is not). To control costs, we batch FCM calls, use DynamoDB on-demand pricing (which is cost-effective for spiky workloads), and use spot instances for non-critical delivery workers (email, SMS). Conclusion This architecture provides a robust, scalable, and low-latency notification system suitable for a 50-million-user e-commerce platform. By leveraging Kafka as the central nervous system, DynamoDB for fast user data access, and channel-specific delivery workers that auto-scale independently, the system meets all stated requirements while remaining operationally manageable and cost-effective. The design is extensible: adding a new notification channel (e.g., WhatsApp) requires only a new delivery worker consuming from a new Kafka topic, with no changes to the upstream event producers or the core processing logic.