System Design: Real-Time Notification Service

REAL-TIME NOTIFICATION SERVICE — SYSTEM DESIGN PLAN

1. REQUIREMENTS RECAP AND CAPACITY ESTIMATION

Traffic math:

• 100M DAU x 10 events/day = 1B notification-triggering events per day.
• Average = 1B / 86,400 ≈ 11,600 events/sec. Peak at 5x ≈ 58,000 events/sec.
• Fan-out multiplier: most events (like, comment, follow, DM) target exactly one recipient, so fan-out is ~1:1. However, events from high-follower accounts (a celebrity posting) can fan out to millions of recipients; treat these as a separate "bulk fan-out" path. Assume effective peak delivery of 80,000–100,000 notifications/sec including bulk campaigns.
• Storage: history of last 100 notifications per user. Assume ~400 bytes per row -> 100 x 400B = 40KB/user -> 100M users ≈ 4TB of hot history, plus replication factor 3 ≈ 12TB. Trivially shardable.
• Connections: assume 20–25% of DAU concurrently online at peak ≈ 20–25M live WebSocket connections. With ~100K connections per gateway node (tuned Linux, epoll), that is 200–250 nodes plus headroom.

Non-functional targets:

• End-to-end p99 latency under 2s for online users (measured from event ingestion to client ack).
• 99.9% availability = ~43 minutes downtime per month.
• At-least-once delivery with client/server-side idempotent deduplication.

2. HIGH-LEVEL ARCHITECTURE

The design is an event-driven pipeline with a clear split between the ingestion path (fast, durable, write-optimized) and the delivery path (channel-specific, retryable).

Flow:
Producer services (Post Service, Social Graph Service, Comment Service, Messaging Service)
-> Notification API / Gateway (gRPC + REST, auth, rate limiting, schema validation, idempotency key)
-> Kafka topic notification.events (partitioned by recipient_user_id where known, else by actor_id)
-> Fan-out Service (resolves recipients, expands celebrity events, applies aggregation rules)
-> Kafka topic notification.deliveries (one message per recipient)
-> Preference & Policy Service (in-line lookup via cache: channel opt-ins, quiet hours, mute/block, digest vs instant)
-> Router / Dispatcher writes to per-channel topics:
deliver.websocket, deliver.mobile_push, deliver.web_push, deliver.email
-> Channel Workers:
WebSocket Dispatcher -> Connection Registry lookup -> Realtime Gateway node -> client
Mobile Push Worker -> APNs (iOS) / FCM (Android)
Web Push Worker -> VAPID/Web Push protocol via browser push services
Email Worker -> SES / SendGrid, with template rendering and digest batching
-> In parallel, a History Writer consumer persists every delivery to the Notification History store and increments unread counters.

Cross-cutting: Template Service, Device Registry, Connection Registry, Dedup Store, Dead Letter Queues, Observability stack, and an Admin/Analytics plane consuming the same Kafka streams.

3. KEY COMPONENTS AND RESPONSIBILITIES

Notification API Gateway

• Single entry point for internal producers (gRPC, protobuf schemas in a registry) and for client read APIs (REST/GraphQL for history, mark-as-read, preferences).
• Responsibilities: authentication (mTLS service-to-service, JWT for clients), per-tenant and per-producer rate limiting/quotas, request validation, idempotency key acceptance, and immediate durable append to Kafka. It returns 202 Accepted quickly; it does no delivery work inline.

Event Ingestion / Kafka

• Durable, replayable log. notification.events with ~256 partitions, replication factor 3, min.insync.replicas=2, acks=all, retention 7 days for replay and incident recovery.
• Provides natural back-pressure and buffering for the 5x peak, decoupling producers from slow third-party providers.

Fan-out Service

• Resolves an event into a recipient list. For 1:1 events, it is a pass-through. For 1:N events (new post to followers, group thread activity), it pages the Social Graph Service and emits recipient messages in batches.
• Hybrid fan-out strategy: push-based (write per recipient) for normal accounts; pull-based/lazy for accounts above a follower threshold (e.g., 1M+), where a single "feed marker" is written and recipients materialize on read. This prevents a celebrity event from creating a 50M-message write storm on the hot path.
• Rate-limits bulk expansions onto a separate low-priority Kafka topic so they never starve interactive notifications.
• Applies aggregation/coalescing: "Alice and 24 others liked your post" instead of 25 rows, using a short tumbling window (e.g., 30–60s) keyed by (recipient, post_id, type) in Redis.

Preference & Policy Service

• Stores and serves per-user channel preferences, category-level opt-ins, quiet hours with timezone, device-level settings, mute/block lists, and legal consent flags (GDPR/CAN-SPAM).
• Read path is cached in Redis with write-through invalidation; p99 lookup target under 5ms. Source of truth in a relational store.
• Enforces frequency capping (e.g., max N pushes/hour per user) to protect user experience and provider quotas.

Router / Dispatcher

• Maps an approved delivery into channel-specific work items. Decides "in-app + WebSocket only if online; otherwise mobile push" using the Connection Registry, and schedules email digests instead of instant sends when preferences say so.

Realtime Gateway (WebSocket layer)

• Stateful tier holding persistent WebSocket connections (fallback: Server-Sent Events, then long polling).
• On connect: authenticate, register (user_id, device_id) -> gateway_node_id in the Connection Registry (Redis Cluster, TTL + heartbeat refresh), then push any unread backlog.
• On delivery: dispatcher looks up node, sends over an internal gRPC stream or a per-node Kafka/Redis Pub/Sub channel; the gateway writes to the socket and waits for a client ack.
• Heartbeats every 30s; missed heartbeats evict the registry entry and future notifications automatically fall back to mobile push.

Channel Workers

• Mobile Push Worker: batches to APNs (HTTP/2 multiplexed, token-based auth) and FCM. Handles per-provider concurrency limits, exponential backoff with jitter, and prunes invalid/unregistered device tokens from the Device Registry.
• Web Push Worker: Web Push protocol with VAPID keys and payload encryption.
• Email Worker: renders templates, supports instant, hourly, and daily digests; handles bounces/complaints via provider webhooks and maintains a suppression list.
• All workers are idempotent consumers keyed by notification_id and write terminal status back to a delivery.status topic.

History Writer & Read Path

• Consumer that persists notifications and maintains unread counters in Redis (authoritative counter periodically reconciled from the store).
• Read API serves the last 100 notifications from the history store, with a Redis cache of the first page per user for sub-10ms typical reads.

Supporting services

• Template Service: versioned, localized templates with variable interpolation; decouples copy changes from code deploys.
• Device Registry: device_token, platform, app_version, locale, timezone, last_seen.
• Dedup Store: Redis with a 24h TTL on (producer_id, idempotency_key) to make at-least-once behave as effectively-once.
• Scheduler: for delayed/scheduled notifications and digest windows (time-bucketed queues in Redis sorted sets or a dedicated scheduler like a Kafka delay topic ladder).
• DLQ + Replayer: poison messages parked and replayable after fixes.

4. DATA MODEL AND DATABASE CHOICES

a) Notification History — Cassandra (or ScyllaDB / DynamoDB)
Table: notifications_by_user

• Partition key: user_id
• Clustering key: created_at DESC, notification_id
• Columns: type, actor_id(s), target_type, target_id, aggregated_count, preview_text, image_url, deep_link, read_at, channels_sent, created_at
• TTL: 90 days; application caps reads at 100 rows.
  Justification: the access pattern is a single, well-known partition key with a time-ordered slice — exactly Cassandra's sweet spot. It offers linear write scalability (we need 60–100K writes/sec sustained), multi-region masterless replication for availability, tunable consistency (write QUORUM/LOCAL_QUORUM, read LOCAL_ONE for the feed), and native TTL for retention. We do not need joins or multi-row transactions here, so a relational store would only add sharding pain and write-amplification risk.

b) Unread counters and hot first page — Redis Cluster

• unread:{user_id} integer counter; notif:page0:{user_id} cached JSON list.
  Justification: counters are read on every app open (extremely high QPS, low value per read) and must be single-digit-millisecond. Redis absorbs this cheaply; Cassandra counters are comparatively costly and error-prone. Counters are reconciled asynchronously so drift self-heals.

c) User preferences and device tokens — PostgreSQL (sharded by user_id) with Redis read-through cache
Tables: user_preferences(user_id, category, channel, enabled, quiet_hours_start, quiet_hours_end, timezone, updated_at), devices(device_id, user_id, platform, push_token, locale, last_seen, active), suppression_list(email, reason, created_at).
Justification: this data is low-volume, read-heavy, relational (users -> devices -> per-category settings), and benefits from transactions and constraints for correctness and auditability (consent records have compliance weight). Volume is small enough (~100M rows) to shard trivially and to cache almost entirely.

d) Connection Registry — Redis Cluster

• conn:{user_id} -> set of {device_id, gateway_node_id, connected_at}, TTL 90s refreshed by heartbeat.
  Justification: ephemeral, extremely high churn, must be fast; durability is unnecessary because a lost entry degrades gracefully to push fallback.

e) Dedup / idempotency — Redis with TTL, backed by nothing (loss only risks a rare duplicate).

f) Templates and config — PostgreSQL + object storage for assets, cached at the edge.

g) Analytics — events streamed from Kafka to a data lake (S3/Parquet) and warehouse (Snowflake/BigQuery) for delivery-rate, open-rate, and latency reporting; ClickHouse for near-real-time operational dashboards.

5. TECHNOLOGY STACK RECOMMENDATIONS

• Language/runtime: Go for the gateway, realtime gateway, and workers (goroutines and low memory per connection suit millions of sockets); Java/Kotlin acceptable for Kafka Streams-heavy aggregation.
• Messaging: Apache Kafka (managed: MSK/Confluent) as the backbone; separate topics per channel and per priority class. Kafka Streams or Flink for windowed aggregation/coalescing.
• Realtime transport: WebSocket over TLS with SSE and long-poll fallbacks; NLB/L4 load balancing with connection draining; sticky routing not required because the registry stores node identity.
• Caching: Redis Cluster (or Elasticache/MemoryDB) for counters, preferences, registry, dedup, rate limits.
• Datastores: Cassandra/ScyllaDB for history; PostgreSQL (Aurora) for preferences/devices; S3 + warehouse for analytics.
• Push providers: APNs, FCM, Web Push (VAPID); email via SES with SendGrid as a secondary provider behind a provider-abstraction layer for failover.
• Infrastructure: Kubernetes with HPA/KEDA scaling on Kafka consumer lag (not just CPU), Envoy/Istio service mesh for mTLS and retries, Terraform for IaC.
• Observability: OpenTelemetry tracing (trace id propagated from producer event to client ack), Prometheus + Grafana metrics, structured logs in Loki/ELK, PagerDuty alerting on SLOs.
• Resilience libraries: circuit breakers and bulkheads per external provider, token-bucket rate limiters, exponential backoff with jitter.

6. SCALABILITY, LATENCY, AND AVAILABILITY STRATEGIES

Scalability

• Every stateless component (API, fan-out, router, workers) scales horizontally; Kafka partition count is the parallelism ceiling, so provision partitions for 5–10x current peak from day one (repartitioning is operationally painful).
• Autoscale on consumer lag with KEDA so a 5x spike triggers scale-out within seconds; keep a warm buffer of 30–40% headroom because scaling is not instantaneous.
• Shard by user_id consistently across Cassandra, Postgres, and Redis so a single user's data is co-located and hot-partition mitigation is uniform.
• Priority lanes: interactive notifications (DM, comment on your post) use a high-priority topic with dedicated consumer groups; bulk/marketing/celebrity fan-out uses a throttled low-priority lane. This guarantees the 2s SLO for the notifications users actually notice.
• Celebrity/hot-key handling via the hybrid push/pull fan-out described above, plus salted partition keys for extremely hot targets.

Low latency (under 2s p99)

• The critical path is intentionally short: API append -> Kafka -> fan-out -> preference cache hit -> registry lookup -> WebSocket write. All lookups are Redis (sub-5ms); no synchronous database writes block delivery.
• History persistence and analytics are asynchronous, off the delivery path.
• Kafka producers tuned with linger.ms=5 and compression (lz4) to balance batching against latency; consumers use manual commit after processing.
• Multi-region deployment with users pinned to the nearest region to cut RTT; WebSocket connections terminate at regional edges.
• Aggregation windows are configurable per type and disabled for latency-critical types like DMs.
• Continuous synthetic probes measure true end-to-end latency per region and per channel.

High availability (99.9%+)

• No single points of failure: multi-AZ for every tier, Kafka RF=3 with min.insync.replicas=2, Cassandra RF=3 with LOCAL_QUORUM writes, Postgres with synchronous standby and automated failover.
• Active-active multi-region for the realtime and delivery tiers; Cassandra replicates cross-region asynchronously, Postgres uses regional read replicas with a designated write region.
• Graceful degradation ladder: if the WebSocket path is unhealthy, fall back to mobile push; if the preference cache is down, fall back to Postgres and then to conservative defaults; if Cassandra writes fail, keep delivering in real time and replay history writes from Kafka afterward.
• Retries with exponential backoff plus jitter, capped attempts, then DLQ with alerting and a replay tool.
• Circuit breakers per external provider so an APNs outage cannot exhaust worker threads and stall email.
• At-least-once delivery plus notification_id dedup on server and client; clients also dedup on reconnect when replaying the unread backlog.
• Reliability practices: chaos/game-day exercises (kill a gateway node and verify push fallback), load tests at 5x peak, blue-green and canary deploys, feature flags for per-channel kill switches, backpressure shedding of low-priority traffic before high-priority.

7. BOTTLENECKS AND TRADE-OFFS

Likely bottlenecks

• Third-party providers (APNs/FCM/email): the hardest ceiling, since throughput is not ours to scale. Mitigate with connection pooling over HTTP/2, batching, per-provider rate limiters, multi-provider failover for email, and queue-based smoothing of spikes.
• Celebrity fan-out: a single post can generate tens of millions of deliveries. Mitigated by hybrid push/pull, throttled bulk lanes, and aggregation.
• WebSocket connection tier: memory and file descriptors per node, plus thundering-herd reconnects after a deploy or network blip. Mitigate with tuned kernel limits, staggered reconnect with jitter and exponential backoff enforced client-side, and slow connection draining during deploys.
• Redis hot keys: a viral post's aggregation key or a shared counter. Mitigate with key salting, local in-process caches with short TTLs, and client-side sharding.
• Cassandra write amplification and compaction pressure at 100K writes/sec. Mitigate with time-window compaction (TWCS) suited to time-series TTL data, and by not storing low-value notification types in history at all.
• Kafka partition skew if partitioning by a low-cardinality key; always partition by recipient_user_id on the delivery topic.

Explicit trade-offs

• At-least-once over exactly-once: exactly-once across heterogeneous external providers is impractical and costly. We accept rare duplicates and solve them with idempotency keys and client-side dedup — cheaper and far more available.
• Eventual consistency for counters and history over strong consistency: an unread badge that is momentarily off by one is acceptable; multi-second latency to guarantee it is not. Reconciliation jobs bound the drift.
• Polyglot persistence over a single database: this adds operational surface (three datastores plus Redis) but each workload gets the right engine. A single Postgres cluster would become the bottleneck at 1B writes/day; a single Cassandra cluster would make preferences and consent auditing awkward.
• Push-based fan-out for most users, pull-based for celebrities: added code complexity, but it is the only way to avoid unbounded write storms.
• Aggregation improves user experience and cuts delivery volume dramatically, but adds up to a window's worth of latency for aggregatable types. We keep windows short and exempt DMs.
• Multi-region active-active raises cost and introduces cross-region consistency subtleties, justified by the availability and latency targets. If budget were constrained, a single-region-with-warm-standby posture would still meet 99.9% but with a longer RTO.
• Kafka over a simpler managed queue (SQS): more operational burden, but we need replay, ordered per-key partitions, multiple independent consumer groups, and 7-day retention for incident recovery.
• Storing only the last ~100 notifications with a 90-day TTL trades archival completeness for cost and read performance; long-term data lives in the analytics lake if ever needed.

8. ROLLOUT PLAN (PHASED)

Phase 1: Ingestion API + Kafka + history writer + read API (in-app notifications only). Establishes the durable backbone.
Phase 2: Realtime gateway, connection registry, unread counters, WebSocket delivery with push fallback.
Phase 3: Mobile push and web push workers, device registry, preference service and quiet hours.
Phase 4: Email instant + digests, template service, suppression handling.
Phase 5: Aggregation/coalescing, hybrid celebrity fan-out, frequency capping.
Phase 6: Multi-region active-active, chaos testing, 5x load validation, SLO dashboards and error budgets.

9. KEY METRICS AND SLOs

• End-to-end p50/p95/p99 latency per channel (SLO: p99 under 2s for WebSocket).
• Delivery success rate per channel; provider error rates and token invalidation rate.
• Kafka consumer lag per topic (primary autoscaling and paging signal).
• WebSocket connection count, churn rate, and ack rate.
• Unread counter drift detected by reconciliation.
• DLQ depth and age.
• Availability per API measured from the client's perspective, tracked against a monthly error budget.

High-level architecture:

The system is event-driven. Product services such as Follow Service, Post Service, Comment Service, Reaction Service, and Direct Message Service publish notification-triggering events to a durable message bus. A Notification Pipeline consumes these events, enriches them, applies user preferences and deduplication rules, persists notification records, and routes delivery jobs to channel-specific delivery workers for real-time websocket/web push, mobile push, and email.

Core flow:

1. A user action occurs, for example user A likes user B’s post.
2. The source service writes its own business transaction, then publishes a notification event to a message broker using an outbox pattern or transactional event log.
3. Notification Ingestion validates the event, normalizes it into a common schema, and assigns an idempotency key.
4. Fan-out and Routing determine recipients, notification type, priority, channel eligibility, and whether the recipient is online.
5. Notification Storage writes the notification to the recipient’s notification history.
6. Real-time delivery is attempted through websocket connections for online web/app users and mobile push for mobile devices where appropriate.
7. Email delivery is queued separately, usually with batching and lower priority to avoid spamming users.
8. Delivery status, retries, failures, and metrics are tracked asynchronously.

Scale estimate:

100 million DAU times 10 events per day equals 1 billion notification-triggering events per day. Average event rate is about 11,600 events per second. Peak load at 5x is about 58,000 events per second. The design should provision for higher internal fan-out because one event may notify multiple recipients, especially comments, mentions, group conversations, or live interactions. A safe initial target might be 100,000 to 300,000 notification deliveries per second at peak, depending on product behavior.

Key components and responsibilities:

1. API Gateway
   Accepts external API requests for reading notification history, marking notifications as read, updating notification preferences, registering devices, and opening websocket connections. Handles authentication, rate limiting, request routing, and basic abuse protection.

2. Event Producers
   Existing product services generate domain events such as UserFollowed, PostLiked, CommentCreated, UserMentioned, DirectMessageCreated, and MessageReactionAdded. Producers should not synchronously call notification delivery services because that would couple product latency to notification infrastructure. They publish to a durable event bus.

3. Event Bus or Message Queue
   A distributed log such as Apache Kafka or Apache Pulsar is recommended. It provides high throughput, durable storage, replay capability, partitioning, consumer groups, and backpressure handling. Topics can be separated by event family or priority, for example social-interactions, direct-messages, notification-delivery-high, notification-delivery-normal, notification-delivery-email.

4. Notification Ingestion Service
   Consumes raw product events, validates schemas, filters invalid/self-notifications where applicable, normalizes event payloads, generates notification IDs, and performs idempotency checks. It also enriches events with lightweight metadata such as actor name, actor avatar reference, post ID, and target object type. Heavy enrichment should be minimized or done asynchronously to protect latency.

5. Fan-out Service
   Determines recipients and creates per-recipient notification tasks. For one-to-one events like a like, follow, or direct message, fan-out is simple. For events involving multiple recipients, such as mentions, group messages, or comment threads, fan-out may produce many delivery records. It should support both fan-out-on-write and fan-out-on-read depending on scale.

Recommended approach:
For normal notifications, use fan-out-on-write and store per-recipient notification records. This makes history retrieval fast and enables unread counts.
For very high fan-out entities, such as celebrity broadcasts or massive groups, use hybrid fan-out: store a shared notification object and materialize only for active users or on read.

6. Preference and Policy Service
   Stores and evaluates user notification preferences, privacy settings, mute/block relationships, quiet hours, channel permissions, locale, email opt-in status, and platform-specific settings. It returns channel eligibility and priority. Preferences should be cached aggressively.

7. Notification Store Service
   Persists notification history and state. It stores the last N notifications, unread/read state, timestamps, type, actor, entity references, and rendering metadata. It supports queries such as latest 100 notifications for a user, unread count, mark one as read, mark all as read, and delete/hide.

8. Realtime Connection Gateway
   Maintains websocket or server-sent event connections for online web and mobile app users. It stores connection presence in a distributed presence store. Delivery workers route real-time notifications to the gateway node holding the recipient’s active connection. For mobile apps in background, delivery falls back to APNs/FCM.

9. Presence Service
   Tracks whether a user is online, which devices are connected, and which gateway node owns each connection. Presence data is ephemeral and should be stored in Redis Cluster or another low-latency distributed cache with short TTL heartbeats.

10. Channel Delivery Services
    Separate workers deliver notifications by channel:
    Mobile Push Worker: Sends to APNs for iOS and FCM for Android. Handles token invalidation, provider errors, retryable failures, and collapse keys.
    Web Push Worker: Sends browser push through the Web Push protocol for users with registered browser subscriptions.
    Websocket Worker: Sends low-latency in-app notifications to online users through the Realtime Gateway.
    Email Worker: Sends email through a provider such as SES, SendGrid, or an internal MTA. It should support batching, templates, unsubscribe rules, and lower priority queues.

11. Device Token Service
    Stores mobile device tokens, browser push subscriptions, device metadata, app version, platform, locale, and last-seen timestamp. It handles token rotation and invalid token cleanup.

12. Template and Localization Service
    Renders notification text and email templates based on notification type, locale, actor, object metadata, and client capabilities. Prefer storing structured notification data and rendering at read time for in-app history, while rendering final payloads at delivery time for push/email.

13. Deduplication and Aggregation Service
    Prevents spam and duplicate notifications. Examples: aggregate “Alice and 12 others liked your post” rather than sending 13 independent push notifications. Use idempotency keys such as event_type + actor_id + recipient_id + object_id + event_time_bucket. Aggregation can be done with Redis sorted sets/counters and then persisted.

14. Metrics, Logging, and Alerting
    Collects end-to-end latency, queue lag, delivery success rate, provider error rate, websocket connection counts, notification creation rate, fan-out factor, database latency, unread count accuracy, and retry/dead-letter queue volume.

Data model:

1. Notification event schema
   notification_event_id: globally unique ID
   source_event_id: ID from source service
   event_type: follow, like, comment, direct_message, mention
   actor_user_id: user who caused the event
   recipient_user_ids or recipient resolver reference
   target_type: post, comment, user, message, conversation
   target_id: target object ID
   created_at: event time
   metadata: compact JSON for additional context
   idempotency_key: stable key for duplicate prevention
   priority: high, normal, low

2. Notification history record
   user_id: recipient partition key
   notification_id: time-sortable unique ID, for example Snowflake/UUIDv7
   notification_type
   actor_user_id or actor summary
   target_type
   target_id
   aggregation_key
   summary_text or render parameters
   created_at
   read_at nullable
   seen_at nullable
   status: created, delivered, failed, hidden
   channels_attempted
   metadata

For history storage, use a wide-column database such as Apache Cassandra, ScyllaDB, or DynamoDB. Partition by user_id and cluster by created_at descending or notification_id descending. This matches the primary query pattern: fetch latest 100 notifications for one user. It scales horizontally, supports high write throughput, and has predictable low-latency reads. Use TTL or background compaction to retain recent history according to product requirements, while keeping at least the latest 100. If strict “last 100 only” is required, maintain a per-user trim job or use TTL plus periodic compaction.

Example logical table:
UserNotifications
partition key: user_id
clustering key: created_at descending, notification_id
columns: type, actor_user_id, target_type, target_id, metadata, read_at, aggregation_key, status

3. Unread count model
   Use Redis for fast unread count reads and writes, backed by durable storage in Cassandra/DynamoDB. Increment on notification creation, decrement or reset on read/mark-all-read. Because counters can drift, periodically reconcile from durable storage or use a read marker model.

Alternative read marker approach:
Store last_read_timestamp per user and treat notifications newer than that as unread. This makes “mark all as read” cheap. For per-notification read state, store read_at on individual records. A hybrid is often best.

4. User preferences
   Use a relational database such as PostgreSQL or MySQL for durable preference records because preferences require consistency, structured updates, and occasional joins/admin operations. Cache hot preferences in Redis or Memcached. For very large scale, shard by user_id or use DynamoDB if the organization prefers managed key-value scale.

Preference schema:
user_id
notification_type
channel: push, web, email, in_app
enabled: boolean
quiet_hours
frequency: immediate, digest, never
updated_at

5. Device tokens and push subscriptions
   Use DynamoDB, Cassandra, or a sharded relational store keyed by user_id and device_id. Token lookups must be fast and highly available. Store token_hash/token, platform, app_version, locale, enabled, last_seen_at, and invalidated_at.

6. Presence data
   Use Redis Cluster with TTL:
   user_id -> active connection IDs, device IDs, gateway node IDs, last heartbeat
   connection_id -> user_id, gateway node, expiry
   Presence can be eventually consistent because it is only an optimization for real-time delivery.

7. Delivery status and audit logs
   Use Kafka topics plus a lower-cost analytical store such as S3/object storage, ClickHouse, BigQuery, or Elasticsearch/OpenSearch for operational debugging, analytics, and delivery reporting. Do not put high-volume delivery logs in the primary transactional store unless required.

Technology stack recommendations:

Message broker: Kafka or Pulsar for durable event streaming, partitioned by recipient_user_id where ordering per recipient matters. Use separate topics for ingestion, fan-out, channel delivery, retries, and dead-letter queues.

Databases:
Cassandra/ScyllaDB/DynamoDB for notification history.
PostgreSQL/MySQL or DynamoDB for user preferences.
Redis Cluster for presence, unread count cache, idempotency short-term cache, rate limits, and aggregation windows.
Object storage plus analytical database for logs and historical analytics.

Realtime transport:
WebSocket for in-app and web real-time notifications. Server-Sent Events can be used for web-only one-way delivery, but WebSocket is more flexible for acknowledgements and heartbeat.

Push providers:
APNs for iOS, FCM for Android, Web Push for browsers. Abstract providers behind a Push Delivery Service to isolate provider-specific behavior.

Email:
Amazon SES, SendGrid, Mailgun, or internal email infrastructure. Use dedicated queues, templates, suppression lists, and digesting.

Compute:
Stateless services on Kubernetes or similar orchestration. Horizontally autoscale consumers based on queue lag, CPU, and delivery latency. Use regional deployment with load balancers and service discovery.

IDs:
Use UUIDv7, ULID, or Snowflake IDs for sortable notification IDs. Ensure idempotency via source_event_id and notification-specific idempotency keys.

Scalability strategy:

1. Partitioning
   Partition Kafka topics by recipient_user_id for per-user ordering where needed. For source events with unknown recipient, partition by source entity until fan-out, then repartition by recipient. Partition notification history by user_id to optimize latest notification queries.

2. Horizontal scaling
   All pipeline services should be stateless except gateways and storage. Consumers can be scaled by increasing topic partitions and worker replicas. Realtime gateways scale by connection count and bandwidth.

3. Backpressure
   If downstream providers slow down, queues absorb spikes. Use separate queues per channel and priority so email delays do not affect direct messages or in-app notifications. Apply rate limits per user, per actor, per notification type, and per provider.

4. Caching
   Cache preferences, device tokens, presence, templates, and unread counts. Use short TTLs for data that changes frequently. Cache misses should not block all delivery; if preferences are temporarily unavailable, fail safely according to product rules, often by sending only mandatory in-app records and deferring external channels.

5. Aggregation
   Reduce fan-out and push volume by aggregating likes and similar interactions. For example, store each like event if needed for analytics, but send one notification per post per time window: “Alice, Bob, and 10 others liked your post.”

6. Hybrid fan-out
   For high-fanout scenarios, avoid writing millions of rows immediately. Store a shared event and materialize notifications for active users first, then lazily for inactive users when they open the app.

7. Multi-region deployment
   Deploy active-active or active-passive across regions. For 99.9% uptime, active-active for realtime gateways and stateless services is recommended, while data stores should use multi-AZ replication at minimum. Global user routing can send users to the closest healthy region. Use regional Kafka clusters with replication or a managed multi-region streaming system.

Low-latency strategy:

1. Keep product services decoupled from notification delivery. Event publication should be fast and reliable.
2. Use Kafka/Pulsar with sufficient partitions and consumers to keep queue lag low.
3. Use Redis for presence lookup and routing decisions.
4. Use websocket delivery for online users because it avoids third-party push provider latency.
5. Persist notification records before or in parallel with delivery depending on reliability needs. A common approach is to write history first, then deliver; for ultra-low latency, delivery and storage can happen concurrently with idempotent retry.
6. Avoid expensive synchronous joins. Events should include enough metadata for rendering, or use cached user/object summaries.
7. Prioritize direct messages and security-sensitive notifications over low-value social notifications.

Reliability and high availability:

1. Durable messaging
   Use Kafka/Pulsar replication across availability zones. Producers use acknowledgements and retries. Consumers commit offsets only after durable processing or after writing idempotent output.

2. Outbox pattern
   Source services write domain changes and outgoing events transactionally to an outbox table. A relay publishes the outbox to Kafka. This prevents lost events when a service succeeds in its database write but fails before publishing.

3. Idempotency
   Every stage must be idempotent. Duplicate events are expected due to retries. Use source_event_id, notification_id, and idempotency_key to avoid duplicate history rows and duplicate push attempts where possible.

4. Retry policy
   Transient failures go to retry topics with exponential backoff and jitter. Permanent failures, such as invalid push tokens, are handled by marking tokens invalid. Poison messages go to dead-letter queues for inspection.

5. Graceful degradation
   If email provider is down, queue email and continue in-app notifications. If push providers are slow, deliver websocket/in-app and retry push later. If preference cache is unavailable, fall back to durable preference reads or conservative defaults. If presence is unavailable, skip websocket and rely on push/history.

6. Replication and backups
   Use multi-AZ databases, regular backups, point-in-time recovery for relational stores, and tested restore procedures. For Cassandra/ScyllaDB, use replication factor 3 across AZs and quorum settings appropriate to latency/reliability trade-offs.

7. Monitoring and SLOs
   Track p50/p95/p99 end-to-end latency from source event creation to client receipt. Alert on queue lag, delivery failure spikes, provider throttling, database hot partitions, websocket disconnect rate, Redis memory pressure, and consumer rebalance storms.

Notification history API:

GET latest notifications:
The client requests the latest 100 notifications. API Gateway authenticates the user, Notification API queries UserNotifications by user_id ordered by created_at descending, enriches any missing display fields from cache, and returns structured records.

Mark notification read:
The API updates read_at for that notification and adjusts unread count idempotently.

Mark all read:
Store last_read_timestamp for the user and optionally asynchronously update older unread rows. This avoids large synchronous writes.

Potential bottlenecks and trade-offs:

1. Fan-out explosion
   Problem: Some events can notify many users, overwhelming queues and storage.
   Mitigation: Hybrid fan-out, priority queues, aggregation, rate limits, and lazy materialization.
   Trade-off: Fan-out-on-read reduces write load but makes reads more complex and can increase read latency.

2. Hot users and hot posts
   Problem: Celebrities or viral posts may generate huge notification volume for one recipient or one object.
   Mitigation: Aggregate notifications by object and time window, shard aggregation keys, and suppress low-value repeated notifications.
   Trade-off: Users may receive less granular notifications.

3. Push provider limits
   Problem: APNs/FCM/email providers can throttle or fail.
   Mitigation: Dedicated provider workers, adaptive rate limiting, retries, token cleanup, and provider-specific backoff.
   Trade-off: External push delivery cannot always meet the under-2-second target; websocket delivery for online users is the reliable low-latency path.

4. Database write volume
   Problem: Per-recipient notification writes at large scale can be expensive.
   Mitigation: Use write-optimized wide-column stores, batch writes, TTL, aggregation, and hybrid fan-out.
   Trade-off: Wide-column stores have limited query flexibility; design must match access patterns.

5. Unread count correctness
   Problem: Cached counters can drift under retries and concurrent read updates.
   Mitigation: Idempotent updates, read markers, periodic reconciliation.
   Trade-off: Exact unread count is more expensive than eventually consistent unread count.

6. Ordering
   Problem: Notifications may arrive out of order due to distributed processing.
   Mitigation: Use sortable IDs and order by event time/notification ID on read. Partition by recipient for stronger per-user ordering where needed.
   Trade-off: Strict global ordering is not practical or necessary.

7. Latency versus durability
   Problem: Writing history before delivery improves reliability but adds latency.
   Mitigation: Use fast local-region storage and parallelize storage/delivery after idempotency assignment.
   Trade-off: Parallel delivery may briefly show a pushed notification before it appears in history if storage is delayed.

8. Preference evaluation complexity
   Problem: Per-event preference checks can be expensive at peak.
   Mitigation: Cache preferences, precompute effective settings, and invalidate cache on preference changes.
   Trade-off: Preference changes may take a few seconds to propagate unless strong consistency is required.

Recommended final design summary:

Use an event-driven architecture with Kafka/Pulsar as the backbone, stateless ingestion/fan-out/routing services, Cassandra/ScyllaDB/DynamoDB for per-user notification history, Redis for presence and hot caches, websocket gateways for sub-2-second online delivery, APNs/FCM/Web Push for platform notifications, and a separate email delivery pipeline. The system should be horizontally scalable, partitioned primarily by user_id, resilient through durable queues and idempotent processing, and optimized for low latency through caching, online presence routing, priority queues, and aggregation. This design meets the functional requirements, supports the expected 1 billion daily events with 5x peaks, and provides a practical path to 99.9% availability with graceful degradation under partial failures.