Design a Real-Time Notification System for a Social Media App

Real-Time Notification System Design for a Social Media App

1. Requirements Recap & Back-of-the-Envelope Estimation

Traffic estimation:

• 10M DAU × 20 notifications/day = 200M notifications/day
• Average rate = 200M / 86,400s ≈ 2,315 notifications/sec
• Peak rate (5x) ≈ 11,500 notifications/sec
• At 100M DAU (future): average ~23,000/s, peak ~115,000/s

Storage estimation:

• Each notification record ~500 bytes (metadata + payload)
• Daily storage: 200M × 500B = 100GB/day → ~36TB/year (before compression/archival)
• History requirement (last 100 per user) is a "hot" read-heavy access pattern, favoring a design where recent notifications are cheap to fetch (e.g., capped list per user) while full history goes to cold storage.

This confirms we're dealing with a write-heavy, fan-out-heavy, latency-sensitive system that must decouple ingestion from delivery.

2. High-Level Architecture

[Event Producers] → [Event Bus / Kafka] → [Notification Service (Consumers)]
                                                    │
                        ┌───────────────────────────┼───────────────────────────┐
                        ▼                            ▼                           ▼
                 [Preference Service]         [Rendering/Templating]     [Rate Limiter/Dedup]
                        │                            │                           │
                        └───────────────┬────────────┴───────────────────────────┘
                                        ▼
                          [Delivery Dispatcher / Router]
                       ┌─────────────┬─────────────┬─────────────┐
                       ▼             ▼             ▼             ▼
                 [Push Service]  [WebSocket/  [Email Service]  [In-App Store
                 (FCM/APNs)]     SSE Gateway]   (SES/SendGrid)   Write - DynamoDB/Cassandra]
                       │             │
                  [Mobile Devices][Connected Clients]

Flow:

1. A source system (e.g., Like Service, Comment Service, Follow Service, Messaging Service) emits an event (e.g., user_liked_post) to a durable message bus (Kafka).
2. The Notification Orchestrator consumes these events, checks user preferences, applies dedup/rate-limiting/batching logic (e.g., "John and 5 others liked your post"), and generates a notification object.
3. The orchestrator persists the notification (for history) and pushes it to a Delivery Dispatcher, which fans out to the appropriate channels based on user preference and device state (online vs offline).
4. Delivery workers handle actual transmission: push notification providers (FCM for Android, APNs for iOS), WebSocket/SSE for in-app real-time badge updates, and email via a transactional email provider.

3. Key Components

3.1 Event Ingestion Layer — Apache Kafka

• All source services publish events to Kafka topics (partitioned by user_id to preserve per-user ordering).
• Kafka provides durability (replication factor 3), high throughput, and natural buffering during traffic spikes — critical since peak traffic is 5x average.
• Topics: notification.likes, notification.comments, notification.followers, notification.messages (or a single topic with event type field, depending on schema evolution needs).

Why Kafka over SQS/RabbitMQ? Kafka handles very high throughput with low per-message overhead and supports replay (useful for reprocessing failed batches or backfilling). SQS is simpler operationally but harder to scale to >100k msg/s cost-effectively and doesn't support consumer-group replay semantics as cleanly.

3.2 Notification Orchestrator Service

• Stateless consumer group reading from Kafka.
• Responsibilities:
  • Preference check: Query a fast key-value store (Redis or DynamoDB) for user notification settings before processing further. This avoids wasted work generating notifications a user has opted out of.
  • Deduplication/Batching: Use a short-lived aggregation window (e.g., Redis sorted sets with TTL) to batch similar events (e.g., multiple likes on the same post within 60 seconds become one notification).
  • Fan-out for followers: For events like "new post from someone you follow," this could require fanning out to millions of followers (celebrity problem). Use a hybrid fan-out model:
    • Fan-out-on-write for regular users (push to each follower's notification feed immediately).
    • Fan-out-on-read for celebrities/high-follower accounts (compute at read time to avoid a write storm).
• Horizontally scalable — scale consumer instances based on Kafka partition count and lag.

3.3 Notification Store (Persistence Layer)

• Primary store: A wide-column NoSQL database like Apache Cassandra or DynamoDB, partitioned by user_id, clustered/sorted by timestamp (descending).
  • This model is ideal because the dominant access pattern is "get last 100 notifications for user X," which is a simple range query on a partition — no joins needed.
  • Cassandra offers tunable consistency and horizontal scalability well past 100M users; DynamoDB offers a fully managed alternative with less operational overhead (trade-off: higher cost at very large scale, and hot-partition risk for extremely active users unless partition keys are salted).
• TTL/Archival: Keep only recent notifications (e.g., 30-90 days) in the hot store; older data archived to cheaper storage (S3 + Glacier) for compliance/audit, with the "last 100" cap enforced at write time (a capped list per user, or trimmed via periodic compaction).

3.4 Delivery Dispatcher

• Reads the finalized notification object and determines channels to use based on:
  • User's channel preferences (push/email/both/none)
  • User's online status (tracked via a presence service backed by Redis, updated by WebSocket/heartbeat)
• Routes to:
  • Push Notification Service: Integrates with FCM (Android) and APNs (iOS). Wrap in an internal abstraction layer to normalize retries, payload formats, and device token management (tokens stored in a user_devices table, refreshed on app launch).
  • In-App Real-Time Delivery: For users actively connected via WebSocket/SSE, push directly to a connection gateway (e.g., a fleet of WebSocket servers behind a load balancer, using something like Socket.IO or a managed service like AWS API Gateway WebSockets). Connection-to-server mapping tracked in Redis so any dispatcher node can find which gateway instance holds a user's connection.
  • Email Service: For less time-sensitive notifications (e.g., weekly digest) or as fallback for offline users on certain notification types, integrate with a provider like Amazon SES or SendGrid, using a separate lower-priority queue since email SLAs are more relaxed (~seconds to minutes is fine).

3.5 Preference Service

• Simple service backed by a relational DB (Postgres) or DynamoDB, cached aggressively in Redis (preferences change infrequently, read extremely frequently — perfect caching candidate).
• Schema: user_id, notification_type, channel, enabled.

4. Data Model

Notification Table (Cassandra/DynamoDB)

Partition Key: user_id
Clustering Key: notification_id (time-based UUID, sorted descending)
Attributes:
  - type (like, comment, follow, message)
  - actor_id (who triggered it)
  - actor_ids (array, for batched notifications)
  - target_object_id (post_id, comment_id, etc.)
  - message_preview
  - created_at
  - read_status (boolean)
  - delivered_channels (array: push, email, in-app)

User Preferences Table

Partition Key: user_id
Attributes: { likes: {push: true, email: false}, comments: {...}, follows: {...}, messages: {...} }

Device Tokens Table

Partition Key: user_id
Clustering Key: device_id
Attributes: platform (ios/android), token, last_active

5. Ensuring Reliability ("No notification lost")

1. Durable messaging: Kafka with replication factor ≥3 and acks=all on producers ensures events aren't lost before processing.
2. At-least-once processing with idempotency: Consumers may reprocess on failure/restart, so notification IDs are deterministically generated (e.g., hash of source event ID + type) to allow idempotent writes — prevents duplicate notifications on retry.
3. Dead-letter queues (DLQ): Failed deliveries (e.g., push provider timeout) go to a DLQ topic with exponential backoff retry (e.g., 3 retries with jitter), then to manual/alerting review if still failing.
4. Write notification to store BEFORE attempting delivery: This decouples "notification exists" (durability/history) from "notification delivered" (best-effort real-time). Even if push delivery fails, the user sees it next time they open the app and poll the notification API.
5. Outbox pattern in source services: To avoid dual-write problems (DB write + event publish), use the transactional outbox pattern so that when a "like" is recorded in the source service's DB, the event is guaranteed to also be published to Kafka via a change-data-capture (CDC) tool like Debezium.

6. Scalability Strategy (10M → 100M DAU)

• Kafka: Increase partition count (partitioned by user_id hash) — scales linearly with more brokers/consumers.
• Notification Orchestrator & Dispatcher: Stateless, horizontally scalable behind consumer groups; scale via Kubernetes HPA based on Kafka consumer lag.
• Cassandra: Add nodes to the ring; consistent hashing distributes load automatically. Watch for hot partitions from viral content/celebrity accounts — mitigate via bucketing (e.g., splitting a celebrity's fan-out across multiple shard keys).
• WebSocket Gateway: Scale horizontally; use sticky sessions via a connection registry in Redis so dispatchers know which gateway node owns which connection, regardless of how many gateway nodes exist.
• Push/Email providers: These are third-party managed services (FCM, APNs, SES) that scale independently; our responsibility is batching and rate-limiting to stay within provider quotas.
• Caching: Aggressive Redis caching of preferences and presence data reduces DB load as user count grows 10x.

7. Key Trade-offs

| Decision | Trade-off |
|---|---|
| Kafka vs. simpler queue (SQS) | Kafka adds operational complexity (needs dedicated ops expertise, ZooKeeper/KRaft management) but provides far superior throughput and replay capability needed at this scale. |
| Cassandra vs. DynamoDB | Cassandra gives more control and potentially lower cost at very large scale but requires in-house ops; DynamoDB is fully managed (faster to build, less ops burden) but can get expensive at extreme scale and has stricter item-size/throughput-partitioning constraints. |
| Fan-out-on-write vs. fan-out-on-read | Write-time fan-out gives lower read latency (great for the 2-second SLA) but risks a "thundering herd" write storm for celebrities; read-time fan-out avoids that but adds latency and computation at read time. The hybrid approach balances both but adds design/code complexity (two code paths). |
| Storing full history vs. capped 100 notifications hot + archive | Reduces hot storage costs and keeps queries fast, but requires an archival/retrieval path for compliance or "load more" features, adding complexity. |
| At-least-once delivery + idempotency vs. exactly-once | Exactly-once semantics in distributed systems are expensive/complex (needs transactional consumers); at-least-once + idempotent writes achieves the same practical guarantee (no duplicate visible notifications) at much lower operational cost. |
| Real-time push for everything vs. batching/aggregation | Aggregating similar notifications (e.g., "10 people liked your photo") reduces notification fatigue and delivery volume, improving both UX and cost, at the expense of slightly more complex orchestration logic and a small buffering delay (still well within the 2-second SLA if window is short, e.g., 5-10s max hold before flush). |

8. Monitoring & Alerting

• Track Kafka consumer lag, delivery success/failure rate per channel, P50/P95/P99 end-to-end latency (event ingestion → device delivery), and DLQ depth.
• SLA dashboards alert on-call if P95 latency exceeds 2s or DLQ growth indicates a systemic delivery failure (e.g., FCM outage), triggering automatic failover logic (e.g., queue holding + retry with backoff rather than dropping).

Summary

The design decouples notification generation (event-driven, Kafka-based, idempotent) from delivery (multi-channel dispatcher with per-channel retry/backoff), backed by a wide-column store optimized for the "recent history per user" access pattern. This separation is what allows the system to meet the 2-second latency SLA while guaranteeing no data loss, and it scales horizontally at every tier (ingestion, processing, storage, delivery) to support 10x future growth with manageable operational and cost overhead.

OVERVIEW

The system should treat every notification as a durable, recipient-specific record and deliver it asynchronously through one or more channels. The durable in-app inbox is the source of truth; mobile push and email are delivery attempts derived from that record. This separation provides low latency without making user-facing social actions wait for external providers such as APNs, FCM, or an email service.

CAPACITY ESTIMATES

At 10 million DAU and 20 notifications per user per day, the system creates about 200 million notifications per day. This is approximately 2,315 notifications per second on average and 11,600 per second at the stated 5x peak. At 100 million DAU, the corresponding rates are about 23,000 per second average and 116,000 per second peak.

These are recipient-notification rates, not merely source-event rates. A celebrity post can produce a hot-key or fan-out spike, so the ingestion and queueing layers should be provisioned above the calculated peak, initially for roughly 25,000 notifications per second and eventually for at least 200,000 per second. Payloads should remain small, with media and full post content referenced by ID rather than embedded.

HIGH-LEVEL ARCHITECTURE

Social services such as Like, Comment, Follow, and Direct Message write their own state and a notification event into a transactional outbox in the same database transaction. Outbox publishers continuously transfer these events to a durable event log such as Apache Kafka. This avoids the dual-write failure in which a social action succeeds but its notification event is lost.

A Notification Processor consumes the source events, validates them, identifies recipients, generates a deterministic notification ID, loads notification preferences, performs lightweight enrichment, and writes the recipient notification into the inbox store. It then publishes one durable delivery command per enabled channel to channel-specific Kafka topics.

Mobile Push Workers consume push commands and invoke APNs or FCM. Email Workers render templates and invoke a provider such as Amazon SES, SendGrid, or an internal mail transfer service. A WebSocket Gateway can also consume in-app delivery commands and immediately update currently connected clients; disconnected clients still see the durable inbox on reconnect. Delivery results and retries are recorded asynchronously.

The main flow is:

Social service and transactional outbox → Kafka source-event topics → Notification Processor → durable inbox store → channel-command topics → WebSocket, APNs/FCM, and email workers.

The read flow is:

Client → API Gateway → Notification API → inbox store and preference store.

CORE COMPONENTS

The API Gateway handles authentication, rate limiting, request routing, and abuse protection. The Notification API supports fetching recent notifications, cursor-based pagination, unread counts, marking one or more items read, and managing preferences.

Kafka provides durable buffering, traffic smoothing, replay, consumer isolation, and horizontal scalability. Topics are replicated across at least three availability zones. Source topics can be partitioned by recipient user ID after recipient expansion, preserving per-user ordering while distributing users across partitions. High-fan-out expansion can use a separate worker pool so one large event cannot block normal traffic.

The Notification Processor should be stateless and autoscaled using queue lag, processing latency, CPU, and throughput. It performs preference evaluation before issuing channel commands. Preferences can be cached in Redis, but the authoritative value remains in a durable store. Cache invalidation events are emitted whenever preferences change.

The inbox store should be a horizontally scalable key-value or wide-column database such as DynamoDB, Cassandra, or ScyllaDB. DynamoDB offers lower operational burden and managed multi-region options; Cassandra or ScyllaDB can be cheaper at sustained large scale but requires more operational expertise. A relational database is less suitable for the primary inbox because write volume, partition growth, and cross-shard scaling would become expensive.

Redis may cache unread counts and the first page of the inbox, but it must not be the system of record. WebSocket gateways are stateless apart from active connection state. A presence directory in Redis or a similar ephemeral store maps users to gateway instances.

DATA MODEL

A notification record contains notification_id, recipient_user_id, type, actor_user_id, object_type, object_id, creation_time, template_version, compact rendering data, read_time, and optional grouping information. Channel state should contain requested channels and delivery status such as pending, dispatched, failed, or suppressed. Large bodies and mutable social content should not be copied into the record unless an immutable snapshot is required.

A suitable inbox key is partition_key = recipient_user_id plus an optional time bucket, and sort_key = reverse_timestamp plus notification_id. Reverse chronological sorting makes the latest page efficient. At ordinary volume, one partition per user is sufficient; exceptionally active accounts can use monthly buckets. The API queries the newest bucket first and follows an opaque cursor into older buckets.

The requirement only exposes the last 100 notifications. The service can retain somewhat more, such as 30 to 90 days, and apply TTL expiration, while the read API returns at most 100. Asynchronously trimming or expiring records is safer and cheaper than performing a synchronous delete on every insert. If strict physical storage of exactly 100 records is required, a background compactor can remove older entries.

Preferences are keyed by user ID and contain per-type, per-channel settings, for example likes.push, likes.email, comments.push, and comments.email, plus global mute, locale, time zone, and a monotonically increasing version. Device tokens are stored separately by user ID and device ID, with platform, token, last-seen time, and validity status. Tokens should be encrypted and invalidated after permanent APNs or FCM errors.

A deterministic notification ID can be derived from source_event_id, recipient_user_id, notification type, and semantic version. The inbox write is conditional on that ID, making replay and duplicate consumption safe. A delivery command has a similarly deterministic ID based on notification ID and channel.

DELIVERY SEMANTICS AND RELIABILITY

The practical guarantee is durable at-least-once processing with idempotent effects. Exactly-once delivery across databases, Kafka, APNs, FCM, and email is not achievable end to end. Transactional outboxes ensure accepted source actions are eventually published. Kafka replication and acknowledged writes prevent queue loss. Conditional inbox writes suppress duplicate records. Channel workers store or otherwise deduplicate delivery attempt IDs, reducing duplicate sends during retries.

A notification is considered accepted only after the source transaction containing its outbox row commits. It is considered durably created after the inbox write succeeds. Kafka offsets are committed only after the corresponding durable write or provider handoff has completed. Transient failures use exponential backoff with jitter. After a bounded number of attempts, commands move to a dead-letter topic with the error, payload reference, and retry history. Operators can replay this topic after remediation.

Mobile providers and email infrastructure cannot guarantee that a device or mailbox presents a notification within two seconds. Therefore, the service should define the latency objective as the time from committed source event to durable inbox visibility and first provider dispatch. A reasonable SLO is 99 percent under two seconds under supported load. APNs and FCM acknowledgments mean provider acceptance, not user display. The durable inbox ensures that provider outages or offline devices do not lose the notification from the user's history.

For currently connected clients, the WebSocket path normally provides the fastest delivery. WebSocket messages carry notification IDs, and clients deduplicate them against fetched inbox records. On reconnect, clients fetch notifications after their last cursor, so dropped socket messages do not create gaps.

SCALABILITY

Kafka topics should begin with enough partitions for expected parallelism and be expanded before reaching partition throughput limits. Recipient ID partitioning spreads ordinary users evenly. Celebrity events should not use actor ID as the partition key because that creates a hot partition. Recipient expansion can divide a large audience into chunks and publish each chunk independently.

Processors, WebSocket gateways, and channel workers are stateless and horizontally scalable. Autoscaling should consider Kafka lag and oldest-message age rather than CPU alone. Separate consumer groups and quotas isolate direct messages from lower-priority likes and email. Under overload, capacity is reserved for direct messages and comments, while email and low-priority likes can queue without being discarded.

Inbox capacity grows linearly with users but remains bounded by TTL and the limited-history requirement. At 100 million DAU and 20 notifications per day, the design must handle roughly two billion daily writes. Time-bucketed user partitions, on-demand or provisioned database capacity, compressed compact records, and no synchronous secondary-index fan-out keep this manageable. Expensive global queries should be served from an analytics pipeline, not the transactional inbox.

HIGH AVAILABILITY AND DISASTER RECOVERY

Every synchronous component runs across at least three availability zones behind health-checked load balancers. Kafka uses replication factor three with strong acknowledgment settings and an appropriate minimum in-sync replica count. The inbox and preference stores use multi-zone replication and point-in-time backups. Deployments use canary or rolling strategies, backward-compatible event schemas, and schema versioning through a registry.

A cost-effective initial design uses one active region with multi-zone redundancy and an asynchronously replicated warm standby region. Recovery procedures promote the standby, restore consumers from replicated offsets or retained events, and replay safely because processing is idempotent. For stricter regional availability, the system can evolve to active-active regional ingestion, with users assigned a home region and globally unique event IDs. Active-active improves recovery time but increases database cost, duplicate handling, ordering complexity, and preference-consistency challenges.

Preference changes should use strongly consistent writes in the user's home region. A notification already durably created before a preference change may still be delivered; this boundary should be documented. For legally required suppression or account deletion, workers should perform an additional authoritative check immediately before external delivery.

LATENCY OPTIMIZATIONS

The critical path contains no synchronous calls to APNs, FCM, or email providers. Event payloads include enough actor and object metadata for basic rendering, avoiding multiple downstream service calls. Missing optional enrichment should produce a generic notification rather than block delivery. Preference records and templates are cached locally or in Redis, and connections to Kafka, databases, and providers are pooled.

Email rendering and dispatch use a separate topic because email is slower and more expensive than in-app delivery. If product requirements allow it, non-urgent likes can be grouped or sent as digest emails, while comments, follows, and direct messages remain immediate. Grouping reduces cost and user fatigue but changes notification semantics, so it must be an explicit product decision.

API DESIGN

GET /v1/notifications?cursor=...&limit=... returns reverse-chronological records, bounded to the latest 100. POST /v1/notifications/read accepts one or more notification IDs or a read-through timestamp. GET and PUT /v1/notification-preferences read and update versioned settings. POST and DELETE device-token endpoints register and revoke devices. All mutation endpoints support idempotency keys.

Unread counts can be maintained as an eventually consistent counter updated from inbox and read events. Since retries can corrupt naive increments, counter updates must be idempotent or periodically reconciled from authoritative records. If exact unread counts are required, the service can query a user-scoped unread index at greater read and storage cost.

OBSERVABILITY AND OPERATIONS

Metrics include event acceptance rate, end-to-end latency by notification type and channel, Kafka lag, oldest queued command, inbox write errors, provider response codes, retry counts, dead-letter volume, WebSocket connection count, preference-cache hit rate, and duplicate suppression rate. Distributed tracing carries source_event_id and notification_id through every stage. Structured logs exclude message bodies and sensitive tokens.

Alerts should be based on SLO burn rates and oldest-message age. Synthetic users continuously generate events and verify inbox, WebSocket, push-provider, and email-provider paths. Dead-letter replay, regional failover, provider outage, Kafka partition loss, and database throttling should be exercised regularly through runbooks and fault-injection tests.

SECURITY AND PRIVACY

Services authenticate with short-lived identities and receive least-privilege access. Data is encrypted in transit and at rest, and device tokens and email addresses receive additional protection. Notification payloads shown on a locked screen should avoid sensitive direct-message content unless the user explicitly enables previews. APIs enforce that users can only access their own inbox and preferences. Retention, account deletion, audit logging, and regional data-residency requirements must be built into the storage lifecycle.

COST AND TRADE-OFFS

Kafka is preferable to synchronous service-to-service calls because it absorbs bursts and enables replay, although it introduces operational complexity and eventual consistency. A managed Kafka service reduces operational risk at higher direct cost. DynamoDB is operationally simple and elastic, while Cassandra or ScyllaDB may lower steady-state storage cost but requires a capable operations team.

Writing the inbox before issuing channel commands ensures history is authoritative and no channel command refers to a missing record, at the cost of a small additional write latency. At-least-once delivery is chosen over an alleged exactly-once design because it remains correct through crashes and external provider retries when paired with deterministic IDs and idempotency.

Recipient-key partitioning preserves per-user order but does not provide global order, which is unnecessary. Strict ordering across every notification would reduce throughput and availability. Similarly, preference caches improve latency but may briefly serve stale values; versioned invalidation and a final check for sensitive suppression balance performance and correctness.

The recommended initial deployment is a managed Kafka cluster, managed DynamoDB-style inbox and preference storage, Redis for ephemeral cache and presence, stateless containerized services on Kubernetes or a managed container platform, APNs/FCM for mobile push, and SES or a comparable email provider. This architecture comfortably supports the current peak, scales horizontally toward 100 million DAU, retains a durable user-visible history, and isolates external delivery failures from the core social application.