Design a Real-Time Collaborative Whiteboard System

# Real-Time Collaborative Whiteboard — System Design ## 1. High-Level Architecture Components and request/data flow: ``` ┌──────────────┐ Web/Mobile/Desktop │ Clients │ (Canvas rendering, local CRDT replica, Clients ─────► │ (100/board) │ WebSocket client, offline buffer) └──────┬───────┘ │ HTTPS (REST) + WSS (WebSocket) ┌──────▼───────┐ │ CDN │ (static assets, exported images) └──────┬───────┘ ┌──────▼───────────────┐ │ Global Load Balancer │ (L7, TLS termination, │ + API Gateway │ auth, rate limiting) └───┬───────────────┬───┘ REST traffic │ │ WS upgrade (sticky by sessionId) ┌──────────────▼──┐ ┌─────▼───────────────┐ │ Stateless App │ │ Realtime Collab │ │ Services │ │ Servers (WS) │ │ (auth, board │ │ - hold in-memory │ │ CRUD, sharing, │ │ board state │ │ exports) │ │ - merge ops (CRDT) │ └───┬────────┬─────┘ │ - broadcast deltas │ │ │ └───┬───────────┬──────┘ ┌───────▼──┐ ┌───▼─────┐ ┌────▼────┐ ┌───▼────────┐ │ Metadata │ │ Object/ │ │ Redis │ │ Session │ │ DB │ │ Blob │ │ Pub/Sub │ │ Routing │ │(Postgres)│ │ Store(S3)│ │+Presence│ │(Consistent │ └──────────┘ └─────────┘ └─────────┘ │ hashing) │ │ └────────────┘ ┌───────▼─────────────┐ ┌─────────────────────────────┐ │ Document/Op Store │ │ Async Workers (Kafka queue) │ │ (DynamoDB/Cassandra: │◄──│ - snapshotting │ │ ops log + snapshots) │ │ - thumbnail/export gen │ └─────────────────────┘ │ - analytics │ └─────────────────────────────┘ ``` **Interaction summary:** Clients authenticate via the API Gateway (REST), then open a persistent WebSocket to a Realtime Collab server. The gateway uses consistent hashing on `sessionId` so that all participants of one board land on the same server (or a small replica set), keeping the authoritative live state in one place. App Services handle non-real-time CRUD (creating boards, sharing, listing, exports). Redis Pub/Sub bridges Realtime servers so that if participants are split across instances, ops still propagate. Async workers periodically persist snapshots and the op log to durable storage. ## 2. Real-Time Communication - **Protocol:** WebSocket (WSS) for full-duplex, low-latency bidirectional messaging. Falls back to HTTP long-polling via a library like Socket.IO for restrictive networks. WebRTC data channels are considered for cursor/presence peer-to-peer, but a server-relayed model is chosen for simplicity and reliability. - **Message model:** Clients send small **operations/deltas** (e.g., `{type:'stroke_add', objId, points, color}`, `{type:'obj_move', objId, dx, dy}`) rather than full board state. The server validates, assigns a sequence/version, merges, and broadcasts the delta to all other session members. - **Fan-out:** Each Realtime server keeps the connection set per board in memory and broadcasts deltas directly. For boards whose members span multiple servers, the originating server publishes the op to a Redis Pub/Sub channel keyed by `sessionId`; subscribed servers re-broadcast to their local connections. - **Presence & cursors:** High-frequency, low-value data (live cursor positions, selections) is throttled (~30–60ms) and sent best-effort, never persisted. - **Latency target (<200ms):** Achieved via regional Realtime clusters, sticky routing (no cross-region hops), tiny binary/compact JSON payloads, and optimistic local rendering (client applies its own op immediately, then reconciles). ## 3. Data Model **Board metadata (Postgres — relational, transactional):** - `boards(board_id, owner_id, title, created_at, updated_at, latest_snapshot_id)` - `users(user_id, name, email, ...)` - `board_permissions(board_id, user_id, role[owner|editor|viewer])` - `sessions(session_id, board_id, started_at, active_user_count)` **Board content (DynamoDB/Cassandra — high write throughput, append-friendly):** - **Op log:** partition key `board_id`, sort key `version` (monotonic). Each row is one operation `{op_type, object_id, payload, user_id, timestamp}`. - **Snapshots:** periodic materialized full-state blobs `{board_id, snapshot_version, state_json/binary}` stored in object storage (S3) with a pointer row. Loading a board = latest snapshot + replay of ops since that snapshot version. **Object structure within a board:** ``` WhiteboardObject { id, type: "stroke" | "text" | "sticky", layer/zIndex, geometry: { x, y, width, height, rotation }, props: { // type-specific stroke: { points:[...], color, thickness }, text: { content, font, color }, sticky: { content, bgColor } }, createdBy, lastModified, version } ``` **Conflict resolution:** Use a **CRDT** (e.g., a list/map CRDT like those in Yjs/Automerge) or OT for the object set, so concurrent edits (two users moving/editing different or same objects) converge deterministically without a central lock. Each object carries a logical clock for last-writer-wins on conflicting property updates. **Large/binary assets** (uploaded images, exported PNG/PDF) live in S3-style blob storage, referenced by URL in the object. ## 4. Scalability and Reliability Strategy **Scaling to 10k sessions / 1M users:** - **Stateless app tier:** Horizontally autoscaled behind the load balancer; trivial to add nodes. - **Realtime tier:** Sharded by `sessionId` via consistent hashing. With 100 users max per board, 10k sessions = up to ~1M concurrent WS connections. A single tuned node handles ~10–50k connections; fan-out is sized so each board's members concentrate on one node. Autoscale on connection count and CPU. - **Redis:** Clustered, used for Pub/Sub fan-out and presence; partitioned by session. - **Databases:** Postgres with read replicas for metadata; DynamoDB/Cassandra scale horizontally for the high-volume op log. Object storage scales effectively infinitely. - **Snapshotting:** Async workers compact the op log into snapshots on a cadence (e.g., every N ops or T seconds) to bound replay cost and storage. **High availability (99.9%):** - Multi-AZ deployment for every tier; multi-region for disaster recovery. - Realtime servers run with a warm standby/replica per board; on node failure, clients auto-reconnect and the new node rehydrates state from the latest snapshot + op log (last few seconds of in-memory ops may be re-sent by clients from their local buffer). - Health checks + automated failover at the load balancer. - Backpressure and rate limiting to protect against op storms. - Durable op log gives a recovery point — no committed work is lost even on server crash. ## 5. Trade-offs **Chosen trade-off: Latency/availability over strong consistency (AP over CP), using CRDTs.** I prioritize sub-200ms responsiveness and offline tolerance by letting clients apply edits optimistically and resolving conflicts with CRDTs/last-writer-wins, accepting **eventual consistency**. The benefit: the UI feels instant, users can edit during brief network partitions, and the system stays available during failovers. The cost: in rare concurrent-edit cases the converged result may not match any single user's intuitive expectation (e.g., simultaneous edits to the same text), and CRDT metadata adds memory/storage overhead. For a whiteboard this is the right call — creative collaboration values fluidity and availability far more than the strict serialized consistency a transactional/locking (CP) approach would impose, which would add round-trip latency and reduce availability during partitions. **Secondary trade-off:** keeping authoritative live state in-memory on a single Realtime node per board (fast broadcasts, simple merge) versus a fully stateless tier (more resilient but higher latency). Mitigated with snapshots + op log so any node can rebuild state on failover.

1. High-Level Architecture A good design is a hybrid of stateless APIs, stateful real-time collaboration workers, durable storage, and an event stream. Main components: - Web/mobile clients - CDN for static assets - API Gateway / Load Balancer - Auth service - Whiteboard metadata service - Real-time collaboration service - Session router / presence service - Message bus / event stream - Persistence pipeline - Primary database for metadata - Object/document store for whiteboard snapshots and element state - Cache (Redis) - Monitoring, tracing, rate limiting Text diagram: Client -> CDN for app assets -> API Gateway / LB -> Auth Service -> Whiteboard API Service -> Session Router -> Real-Time Collaboration Nodes (WebSocket) -> Redis / in-memory session state -> Pub/Sub or Kafka topic per shard -> Persistence workers -> Snapshot Store -> Metadata DB Flow: - User opens app and authenticates. - Client fetches whiteboard metadata and latest persisted state through REST/HTTP APIs. - Client then upgrades to WebSocket for the board session. - Session router sends the client to the collaboration node responsible for that whiteboard. - Users send operations such as draw stroke segment, create text box, move object, edit sticky note. - Collaboration node validates, sequences, and broadcasts operations to all participants in the session. - Operations are appended to an event log and periodically compacted into snapshots. - On reconnect or reopen, client loads latest snapshot plus recent operations. Practical partitioning: - Use consistent hashing or a board-to-node mapping service so one board session is owned by one collaboration node at a time. - This simplifies ordering and conflict handling. - With 10,000 concurrent sessions and up to 100 users/session, the system is large but manageable with horizontal scaling of collaboration nodes. 2. Real-Time Communication Protocol: - WebSocket over TLS is the primary choice. - Reason: low-latency bidirectional communication, broad browser/mobile support, simpler than long polling, and efficient for frequent small updates. Update model: - Client sends operations, not full board state. - Example operations: - start_stroke, append_stroke_points, end_stroke - create_text, edit_text - create_sticky, move_object, resize_object, delete_object - Collaboration node timestamps/sequences operations and broadcasts them to all clients in that board. Ordering and fanout: - Within a single board, maintain a monotonically increasing sequence number. - All operations from that board pass through its owner collaboration node, which provides total order per board. - Broadcast locally to connected clients; if replicas or multiple nodes serve the same board for failover, use Redis Pub/Sub or Kafka for replication/eventing. Latency target under 200ms: - Keep collaboration nodes geographically close to users using regional deployments. - Use sticky routing so all users of a board hit the same node in-region when possible. - Broadcast deltas only, compress payloads, batch pen points every few milliseconds. - For freehand drawing, allow local optimistic rendering immediately on the sender before server ack, then reconcile if needed. Conflict handling: - For object-based edits like move text box or edit sticky note, use per-object versioning. - For highly concurrent state, use operational transform or CRDT-inspired merging if rich multi-user editing is needed. - For a high-level whiteboard system, a simpler model is acceptable: - Pen strokes are append-only and naturally merge well. - Object transforms use last-write-wins or server ordering. - Text content can use simpler whole-object edit locking initially, or later evolve to OT/CRDT for concurrent text editing. 3. Data Model Use a split model: metadata relational DB + board content in document/blob storage + operation log. A. Whiteboard metadata Table: Whiteboard - board_id - owner_user_id - title - created_at - updated_at - last_snapshot_id - access_policy - region - status Table: WhiteboardMember - board_id - user_id - role (owner/editor/viewer) - invited_at Table: ActiveSession - session_id - board_id - user_id - connection_id - joined_at - last_heartbeat - presence_state B. Board content model Represent board as a collection of elements on an infinite or bounded canvas. Element base fields: - element_id - board_id - type: stroke | text | sticky_note | shape - created_by - created_at - updated_at - z_index - position {x, y} - rotation - style object - version - deleted flag Stroke element: - element_id - points: polyline or Bezier-compressed point list - color - width - opacity Text element: - element_id - text_content - font_family - font_size - bounding_box Sticky note element: - element_id - text_content - background_color - bounding_box C. Operation log Table or stream: BoardOperation - op_id - board_id - seq_no - user_id - op_type - target_element_id - payload - client_timestamp - server_timestamp Payload examples: - create element with initial properties - append stroke points - move object from old position to new position - patch text - delete element D. Snapshots Snapshot object stored in document/blob store: - snapshot_id - board_id - base_seq_no - created_at - serialized board state or chunked spatial partitions Why snapshots + operation log: - Replaying entire history forever becomes too slow. - Snapshots allow fast load. - Recent ops after snapshot restore the latest state. Optional optimization for large boards: - Spatial chunking: partition canvas into tiles/regions so clients load only visible content. - Useful if boards become very large. 4. Scalability and Reliability Strategy Scalability A. Horizontal scaling of collaboration nodes - Collaboration service is the critical path. - Each node maintains in-memory active board sessions assigned to it. - Sessions are sharded by board_id. - If average session size is modest, many sessions can be hosted per node. - Load balancer + session router ensure users join the correct node. B. Event-driven persistence - Do not synchronously write every pen point to the primary DB before broadcast; that would hurt latency. - Instead: - collaboration node accepts operation - appends to durable log / replicated queue - broadcasts immediately - async workers persist and compact to snapshots - For durability, use Kafka/Pulsar or a replicated write-ahead log before ack if stronger guarantees are needed. C. Efficient drawing traffic - Pen tool can generate many points per second. - Reduce volume by: - client-side point simplification - batching points every 10-30ms - binary encoding instead of verbose JSON if needed - gzip/permessage-deflate on WebSocket D. Caching - Redis cache for board metadata, presence, routing map, hot snapshots. - Recent boards and permissions can be cached to reduce DB load. E. Storage strategy - Relational DB (PostgreSQL/MySQL) for users, boards, permissions, sharing metadata. - Document/blob store (S3/object storage or document DB) for snapshots. - Stream/log system for operations. - This avoids forcing one database to handle all access patterns. Reliability / High Availability A. Multi-AZ deployment - Run API, collaboration nodes, Redis, and databases across multiple availability zones. - Load balancers health-check instances and reroute automatically. - 99.9% uptime is achievable with multi-AZ and rolling deploys. B. Collaboration node failover - Because board ownership is stateful, node failure is important. - Mitigation: - operations are continuously replicated to durable log - periodic in-memory checkpoints or fast snapshots - session router detects node loss and reassigns board to another node - clients reconnect automatically and reload snapshot + tail ops - Short disruption is acceptable; users recover quickly. C. Presence and heartbeats - WebSocket heartbeats detect disconnects. - Presence service updates active user state with TTLs in Redis. D. Backpressure and protection - Per-user and per-board rate limiting. - Drop or coalesce excessive move/drag events. - Separate control plane traffic from data plane. - Protect against a noisy session consuming all node resources. E. Durability choices Two options: - Lower latency: ack after operation is accepted in memory and replicated asynchronously. - Higher durability: ack only after append to replicated log. I would choose ack-after-log-append for object mutations and batched treatment for pen segments to balance reliability and speed. Capacity reasoning - 10,000 concurrent sessions x average maybe 10 active users/session is 100,000 concurrent connections on average; worst case 1,000,000 if all sessions have 100 users, but that is likely not the intended steady state. Design should support hundreds of thousands of WebSockets horizontally. - Collaboration nodes can each support tens of thousands of idle/light connections or fewer for heavy sessions, so scale out with autoscaling and shard by active board count and event rate. 5. Trade-off Major trade-off: consistency vs. latency. I would prioritize low latency and smooth collaboration over strict global consistency for every action. What this means: - Clients optimistically render their own drawing immediately. - Server provides authoritative ordering per board, but some operations may be merged using simple rules like server order or last-write-wins. - Persistence is near-real-time rather than blocking every update on a database transaction. Why this trade-off is good here: - Whiteboard UX is highly interactive; users notice lag much more than occasional minor reconciliation. - Pen strokes and object moves are tolerant to eventual consistency within a very small window. - Strict serializable transactions for all edits would increase latency and reduce throughput. Cost of this choice: - During concurrent edits to the same object, users may see small jumps or overwritten updates. - Recovering from node failure may replay a short tail of operations. - Rich concurrent text editing may need a more advanced OT/CRDT mechanism later. Summary design - REST APIs for auth, board metadata, snapshots. - WebSockets for live collaboration. - One logical owner collaboration node per board for sequencing and fanout. - Redis/cache for routing and presence. - Kafka/PubSub/event log for durable operation streaming. - Relational DB for metadata/permissions, object store or document store for snapshots and board state. - Multi-AZ deployment, autoscaling, reconnection/failover, and asynchronous snapshotting for reliability and scale. This architecture meets the near-real-time requirement, supports persistence, scales to many active sessions, and keeps the system operationally practical.