Sweaty Imposter

Design Dropbox / File Sync

System DesignSyncStorage HeavySenior3 min readUpdated

Files change on several devices at once, uploads drop halfway, and the same bytes get uploaded again and again. The design splits files into content-addressed blocks and keeps a change log every device can replay.

1 · Requirements

The Prompt

Multi-device file sync with conflict resolution, deduplication, and resumable uploads.

Functional

  • Upload and download files of any size, resuming after a dropped connection
  • Sync changes to every device signed in to the account
  • Keep file revisions and resolve conflicting edits
  • Share files and folders

Non-Functional

  • A change shows up on other online devices within seconds
  • No edit is silently lost when two devices conflict
  • Identical blocks are stored once

2 · Estimates

Drill This Phase First

Show the Answer and Estimates

Answer

About 12,000 a second

100M times 10 is 1B changes a day. 1B divided by 86,400 seconds is about 11,600, so call it 12,000 writes a second.

Changes
100M daily users making 10 file changes a day: 1B changes, about 12,000 writes a second
Blocks
4 MB blocks and about 30% dedup: about 5B blocks a day, 3.5B of them unique
Storage
3.5B unique blocks at up to 4 MB: about 14 PB a day raw, and about 25 PB with erasure coding and a cross-region copy
Metadata
About 100B files at 500 bytes each: roughly 50 TB in SQL, sharded by user_id
Sockets
About 20% of daily users online: 20M WebSocket connections across about 10,000 gateway nodes
CDN
The CDN download path saves about 60% of egress for shared and popular files

3 · API

POST /blocks { sha256, size } → { upload_url } | { already_exists: true }
PUT /blocks/:sha256 (binary) → 200
POST /files/commit { path, version, block_list[sha256...] } → { file_id, rev }
GET /files/:id?rev=… → { manifest, block_urls[] }
GET /changes?cursor=… → { events[], next_cursor }
WS /notify: pushes { file_id, rev, actor } on change

Data Model

files (SQL, sharded by user_id):
  file_id (PK)    : uuid
  user_id         : uuid
  path            : string
  current_rev     : int
  size            : bigint
  modified_at     : timestamp
  deleted         : bool

file_revisions (SQL):
  file_id (FK)    : uuid
  rev             : int
  block_list      : json array of sha256
  author_device   : uuid
  ts              : timestamp

blocks (KV / object store):
  sha256 (PK)     : string
  size            : int
  refcount        : int       (for GC)
  s3_key          : string

4 · High-Level Design

To upload, the client asks which block hashes are new, sends only those, and commits a revision that lists every block. Other devices hear about the revision over WebSocket, fetch the manifest, and download only the blocks they lack.

5 · Deep Dives

Blocks and Dedup

Hash each 4 MB block with SHA-256 and ask the block service which hashes already exist. Only new blocks get uploaded, so a copied file costs one metadata write. Resuming an upload is the same check: send the blocks the server still lacks.

Conflicts

Every commit names the revision it was based on. If the current revision has moved on, the commit is a conflict: keep the server version and save the incoming one as a conflicted copy beside it, so no edit disappears. Vector clocks can detect concurrent edits, and the conflicted copy is what users understand.

The Change Log

Each user's changes form an ordered log with a cursor. A device that was offline for a week replays from its last cursor, and an online device gets a WebSocket nudge to pull. Change data capture from the metadata database keeps the log in step with committed revisions.

6 · Wrap Up

Wrap up with the next risks: cleaning up blocks once no revision points at them, permission checks on shared folders, and a hot shard for one very large team.

The Short Version

Chunk files into ~4MB blocks; SHA-256 dedup. Metadata in SQL. Notify clients via WebSocket or long-poll. Resumable via chunk-level acks.

Building Blocks