Design a Real-Time Ride-Sharing Notification System

High-level architecture (components and interactions) 1) Event producers (source services) - Trip Service: emits ride lifecycle events (driver_assigned, ride_completed). - Dispatch/Matching Service: emits driver_assigned and any reassignment events. - ETA/Location Service: continuously computes ETA from driver GPS streams; emits driver_arriving_soon when ETA threshold (e.g., <=2 minutes) is crossed with hysteresis. - Promotions Service: creates campaign events with geo-targeting rules and audience definitions. 2) Notification Ingestion + Event Bus - All services publish domain events to a durable event bus. - Events are standardized (user_id, ride_id, event_type, timestamp, payload, idempotency_key, priority, locale). 3) Notification Orchestrator (rules + routing) - Consumes events from the bus. - Applies business rules (who to notify: rider, driver; quiet hours; user opt-outs; rate limits; do-not-disturb; fallback channels). - Enriches notifications (fetch driver name/vehicle, receipt link, ETA text) via cached reads. - Produces “notification jobs” to channel-specific queues with priority (transactional > promotional). 4) User/Device & Preferences Service - Stores device tokens (APNs/FCM), platform, app version, last-seen, language, and notification preferences. - Exposes low-latency lookup (cache-first). 5) Delivery Workers (channel adapters) - Push Gateway: sends to Apple APNs and Google FCM. - SMS Gateway (optional fallback for critical messages): Twilio or direct aggregator. - In-app/WebSocket Gateway (optional): for users currently active in the app. 6) Delivery Tracking + Retry + DLQ - Delivery attempts recorded (sent, accepted by provider, failed with reason). - Automatic retries with exponential backoff for transient failures. - Dead-letter queue for poison messages; alerting and replay tools. 7) Promotional Targeting Pipeline - Geo audience builder: converts geographic areas (geohash/H3 cells) + eligibility criteria into target user sets. - Uses near-real-time location signals (last known location) and/or home/work region. - Outputs batches of notification jobs into lower-priority queues with throttling. 8) Observability and Ops - Metrics: end-to-end latency p50/p95/p99, queue lag, provider error rates, token invalidation rates. - Tracing: correlate event_id → job_id → provider request_id. - Admin console: campaign management, replay, suppression lists. Key technology choices and justification 1) Message queuing / event streaming - Apache Kafka (or managed equivalents like AWS MSK / Confluent Cloud) as the central event bus. Justification: high throughput during rush hours, partitioning for horizontal scale, durable log for replay, consumer groups for independent scaling, good fit for at-least-once processing. - Separate topics for: - ride-events (transactional) - eta-events - promo-events - notification-jobs-high (priority) - notification-jobs-low (promo) - delivery-results 2) Datastores - Device tokens and preferences: DynamoDB (or Cassandra) keyed by user_id. Justification: predictable low-latency reads at massive scale, high availability, easy horizontal scaling. - Delivery tracking / analytics: - Hot path: DynamoDB/Cassandra for recent state (last status per notification_id). - Long-term analytics: data lake (S3/GCS) + warehouse (Snowflake/BigQuery) fed by Kafka Connect. - Campaign/audience metadata: Postgres (or Aurora) for relational management (campaigns, schedules, creatives). - Caching: Redis (clustered) for device token lookups, user preference cache, and template fragments. 3) Push notification services - APNs for iOS and FCM for Android. Justification: official, reliable, scalable push infrastructure; supports priority and collapse keys. - Optional SMS provider for fallback to meet reliability for critical transactional notifications. 4) Geo targeting - H3 or Geohash indexing for geographic regions. Justification: efficient mapping from lat/lon to discrete cells; supports querying “users in these cells”. - Stream processing: Kafka Streams / Flink for maintaining “users-in-cell” membership based on location updates. Low latency (<2s) and high reliability (at-least-once) 1) Low latency strategy - Prioritize transactional notifications: - Use dedicated high-priority topics/queues and worker pools. - Apply strict per-message SLAs: short batching windows (or none) for urgent events. - Cache-first enrichment: - Orchestrator reads device tokens/preferences from Redis; fall back to DynamoDB on cache miss. - Keep payload minimal; include deep links to fetch details in-app. - Minimize synchronous dependencies: - Producers publish events asynchronously. - Orchestrator avoids calling multiple microservices in-line; uses precomputed data (e.g., driver info already in event or obtainable from cache). - Connection reuse and provider best practices: - Maintain persistent HTTP/2 connections to APNs; reuse FCM connections. - Use provider priority flags appropriately. - Control “arriving soon” noise: - ETA service emits only on threshold crossing with cooldown (e.g., don’t resend within N minutes) to reduce load and keep latency for critical messages. 2) At-least-once delivery and correctness - At-least-once from Kafka + consumer commits after processing. - Idempotency: - Each notification job carries a deterministic idempotency_key (e.g., user_id + ride_id + event_type + version). - Orchestrator writes a “job created” record (or dedupe key) with conditional put to prevent duplicate job creation on replays. - Delivery workers record attempts keyed by notification_id to avoid double-sending when possible. - Provider-level dedupe: - Use APNs/FCM collapse keys for certain types (e.g., driver arriving soon) to ensure the latest replaces prior. - Retry policy: - Transient failures: retry with exponential backoff and jitter. - Permanent failures (invalid token): mark token invalid, stop retrying. - DLQ for repeated failures; operator workflows for replay. 3) Reliability and availability - Multi-AZ deployment for Kafka, Redis, DynamoDB (managed), and stateless services. - Backpressure: - If push providers degrade, queues absorb spikes; workers scale but capped to avoid provider rate limits. - Exactly-once is not required; at-least-once plus idempotency is sufficient for user-facing notifications. Scaling to handle peak loads 1) Throughput estimation (order-of-magnitude) - 500k rides/day; each ride might generate 2–4 transactional notifications (assigned, arriving soon, completed/receipt) for rider; plus driver-side notifications. - Peaks during rush hour can be 10–20x average. Design for several thousand notifications/sec sustained bursts. 2) Horizontal scaling approach - Kafka partitioning: - Partition by user_id (or ride_id) to preserve ordering per user/ride for related notifications. - Scale partitions to match expected peak consumer parallelism. - Stateless services: - Orchestrator and delivery workers are stateless and auto-scaled (Kubernetes HPA based on CPU + queue lag). - Separate pools and isolation: - Separate topics/queues and worker deployments for transactional vs promotional. - Hard quotas so promotions never starve transactional delivery. 3) Promotional message scaling - Precompute audience: - Campaign expands to H3 cells; fetch eligible users via “users-in-cell” store. - Fanout in batches with throttling; enqueue jobs to low-priority queue. - Rate limiting: - Global and per-region caps; time-slicing across cells. - Respect user-level frequency caps and opt-outs. 4) Data and cache scaling - Redis cluster sized for high QPS reads; use consistent hashing and replication. - DynamoDB/Cassandra with enough read capacity; cache protects it during spikes. - Token invalidation handling to reduce repeated failed sends. Major trade-offs 1) At-least-once vs exactly-once - Trade-off: at-least-once can cause duplicates; exactly-once would add complexity and latency. - Decision: accept at-least-once with strong idempotency keys and collapse keys to minimize user-visible duplicates. 2) Consistency vs availability - Device tokens/preferences may be slightly stale (eventual consistency) if a user changes settings. - Decision: favor availability and low latency; use short TTL caches and versioning to converge quickly. 3) Cost vs performance - Maintaining a real-time geo audience index and stream processor costs more than batch targeting. - Decision: use near-real-time geo targeting only when needed; otherwise allow campaigns with slower batch expansion. Isolate promo workloads to cheaper capacity. 4) Payload enrichment approach - Fully enriched messages require more synchronous reads and can increase latency. - Decision: keep push payload small and rely on deep links; enrich only when data is already in-event or cached. 5) Provider dependencies and fallback - Adding SMS fallback increases cost and compliance scope. - Decision: enable SMS fallback only for critical transactional notifications (e.g., driver assigned/arriving) and only when push repeatedly fails or user has no valid token. Summary This design uses a durable event streaming backbone (Kafka), a rules-based notification orchestrator, and scalable channel delivery workers to achieve sub-2-second transactional notification delivery with at-least-once reliability. It scales horizontally via partitioning and autoscaling, isolates promotional traffic to protect time-sensitive messages, and manages duplicates through idempotency and provider collapse keys while balancing cost and operational complexity.

Real-Time Ride-Sharing Notification System: High-Level Architecture Design Overview The notification system must reliably deliver time-sensitive messages to 1 million daily active users across 500,000 daily rides, with sub-2-second latency and at-least-once delivery guarantees. The design follows an event-driven, microservices architecture with clear separation between event production, routing, delivery, and tracking. --- 1. High-Level Architectural Components and Interactions The system is composed of five major layers: Event Producers: The core ride-matching and trip-management services emit domain events (driver assigned, driver arriving, ride completed, promo triggered) to a central message broker. These services are decoupled from the notification system entirely. Message Broker (Event Bus): A distributed streaming platform ingests all events and fans them out to downstream consumers. This is the backbone of the asynchronous pipeline. Notification Orchestrator Service: A stateless microservice that consumes events from the broker, enriches them with user preferences and device tokens (fetched from a User Profile Store), determines the correct notification channel (push, SMS, in-app), and publishes formatted notification jobs to channel-specific queues. Channel Delivery Workers: Dedicated worker pools for each channel (APNs for iOS, FCM for Android, Twilio for SMS, WebSocket server for in-app). Each worker consumes from its queue, calls the appropriate third-party gateway, and records delivery status. Delivery Tracking and Audit Store: A persistent log of every notification attempt, its status (sent, delivered, failed), and timestamps. This feeds dashboards and retry logic. Flow summary: Ride service emits event → Message broker → Orchestrator enriches and routes → Channel queue → Delivery worker → Push/SMS gateway → Device. Delivery status is written back to the audit store asynchronously. --- 2. Key Technology Choices and Justifications Message Broker: Apache Kafka. Kafka provides high-throughput, durable, ordered event streaming with configurable retention. Its consumer group model allows multiple orchestrator instances to process events in parallel. Kafka's replication factor (set to 3) ensures no event is lost if a broker node fails. For rush-hour spikes, Kafka absorbs bursts without back-pressure on upstream services. Orchestrator and Workers: Kubernetes-deployed Go or Java microservices. Go offers low memory overhead and high concurrency via goroutines, ideal for I/O-bound notification dispatch. Kubernetes enables horizontal pod autoscaling based on Kafka consumer lag metrics. User Profile and Device Token Store: Redis (primary, for hot data) backed by PostgreSQL (source of truth). Device tokens and user preferences are cached in Redis with a TTL. Cache-aside pattern ensures the orchestrator retrieves tokens in under 1 ms on cache hit, avoiding database bottlenecks at scale. Push Notification Gateways: Firebase Cloud Messaging (FCM) for Android and Apple Push Notification service (APNs) for iOS. Both support batch sending APIs. For SMS fallback, Twilio provides reliable global delivery with delivery receipts. In-App / Real-Time Channel: A WebSocket gateway (using Socket.io or a custom server backed by Redis Pub/Sub) maintains persistent connections with active app sessions. For driver-arriving and driver-assigned notifications, the in-app channel is attempted first because it is the lowest latency path. Delivery Audit Store: Apache Cassandra or Amazon DynamoDB. Both are optimized for high write throughput and time-series access patterns. Each notification attempt is written as an immutable record keyed by (user_id, notification_id, timestamp). Geographic Targeting for Promotions: A geospatial index using PostGIS or Redis with the GEO commands stores user last-known locations. A promotional campaign service queries this index to build target user lists, then publishes individual notification events to Kafka, keeping the delivery pipeline uniform. --- 3. Low Latency and High Reliability Strategy Low Latency (under 2 seconds end-to-end): The critical path for transactional notifications (driver assigned, driver arriving) is: event emission → Kafka ingestion (< 10 ms) → orchestrator processing including Redis token lookup (< 50 ms) → FCM/APNs API call (typically 100–400 ms) → device receipt. Total expected p99 latency is well under 2 seconds under normal load. To minimize orchestrator processing time, token enrichment is done via a single Redis pipeline call rather than sequential lookups. The orchestrator is stateless and horizontally scaled, so no single instance is a bottleneck. WebSocket delivery for in-app notifications bypasses push gateways entirely, achieving sub-100 ms delivery for users with active sessions. High Reliability (at-least-once delivery): Kafka consumer offsets are committed only after the orchestrator has successfully published the notification job to the channel queue. If the orchestrator crashes mid-processing, the event is re-consumed and reprocessed. Channel delivery workers use an idempotency key (derived from notification_id) when calling FCM/APNs, preventing duplicate notifications even if a worker retries after a transient failure. A dead-letter queue (DLQ) captures notifications that fail after a configurable number of retries (e.g., 3 attempts with exponential backoff). An alerting system monitors DLQ depth and triggers on-call escalation. For SMS fallback: if push delivery is not acknowledged within 30 seconds (checked via FCM delivery receipts), the system automatically falls back to SMS for critical notifications (driver assigned, ride completed). --- 4. Scaling Strategy for Peak Loads Rush-hour peaks are predictable (morning and evening commutes). The system uses two complementary scaling approaches: Proactive (Scheduled) Scaling: Kubernetes Horizontal Pod Autoscaler (HPA) is pre-warmed before known peak windows using a cron-based scaling policy. Orchestrator and worker replica counts are increased 10 minutes before rush hour. Reactive Scaling: Kafka consumer lag is exposed as a custom metric to the Kubernetes Metrics Server. If lag on the notification topic exceeds a threshold (e.g., 10,000 unprocessed messages), HPA adds orchestrator pods automatically. Worker pools scale similarly based on their queue depths. Kafka partitioning: The notification topic is partitioned by user_id (or ride_id) to ensure ordered processing per user while allowing high parallelism. With 100 partitions, up to 100 orchestrator instances can process concurrently. Promotion notifications, which are bulk and non-time-critical, are processed on a separate lower-priority Kafka topic with a dedicated consumer group. This prevents promotional bursts from competing with transactional notifications for processing resources. Database scaling: Redis is deployed as a cluster with read replicas. PostgreSQL uses connection pooling (PgBouncer) and read replicas for profile lookups. Cassandra scales horizontally by adding nodes. Estimated throughput: 500,000 rides/day with an average of 3 notifications per ride equals 1.5 million transactional notifications per day, or roughly 17 per second on average. During a 2-hour rush-hour peak handling 30% of daily rides, peak throughput reaches approximately 70 notifications per second. This is well within the capacity of a modest Kafka and worker cluster, leaving ample headroom. --- 5. Major Trade-offs At-least-once vs. Exactly-once Delivery: Exactly-once delivery across distributed systems and third-party gateways (FCM, APNs) is practically impossible to guarantee end-to-end. The design accepts at-least-once semantics and mitigates duplicate delivery using idempotency keys. The trade-off is occasional duplicate notifications (e.g., two "driver assigned" messages) versus the complexity and latency cost of distributed transactions. For a ride-sharing app, a rare duplicate is far less harmful than a missed notification. Cost vs. Performance: Maintaining a persistent WebSocket connection for every active user is expensive in terms of server resources. The design uses WebSockets only for users with active app sessions (typically during an active ride), falling back to push notifications otherwise. This reduces infrastructure cost while preserving low latency for the most critical moments. Consistency vs. Availability: The Redis token cache may serve a stale device token if a user reinstalls the app and the cache has not yet been invalidated. The design accepts this eventual consistency window (typically seconds to minutes) in exchange for the availability and latency benefits of caching. A failed push due to a stale token triggers a cache refresh and retry, so the impact is a slightly delayed notification rather than a lost one. Simplicity vs. Flexibility in Channel Routing: A more sophisticated routing engine could apply machine-learning models to choose the optimal channel per user. The current design uses a simple rule-based router (in-app first, then push, then SMS fallback) for predictability, lower operational complexity, and easier debugging. This can be evolved incrementally. Promotion Isolation: Separating promotional traffic onto its own Kafka topic and worker pool adds operational complexity (more topics to monitor, more consumer groups to manage) but is a necessary trade-off to protect transactional notification SLAs from being degraded by bulk promotional sends. --- Conclusion This architecture delivers a robust, scalable notification system by combining Kafka's durable event streaming, Redis-backed low-latency enrichment, multi-channel delivery workers, and Kubernetes-based autoscaling. The design prioritizes sub-2-second delivery for transactional notifications and at-least-once reliability through idempotent retries and dead-letter queues, while isolating promotional traffic to protect core user experience. The trade-offs made favor operational simplicity and availability over theoretical perfection, which is appropriate for a high-velocity consumer product.