Sweaty Imposter

Design a Web Crawler

System DesignWrite HeavyBatchSenior3 min readUpdated

The crawler fetches about 400 pages a second without overloading any one site, and most links it finds point at pages it has already seen. The design rests on a URL frontier that enforces politeness per host and cheap checks that skip duplicate URLs and duplicate content.

1 · Requirements

The Prompt

Crawl 1B web pages a month for a search index. Respect robots.txt and per-site limits, skip duplicate pages, and recrawl pages as they change.

Functional

  • Crawl pages from a seed list and follow the links they contain
  • Respect robots.txt and a request limit for each host
  • Skip URLs already crawled and pages that duplicate stored content
  • Recrawl pages on a schedule that matches how often they change
  • Store raw pages for the indexer

Non-Functional

  • Sustains about 400 pages a second, with headroom for recrawls
  • Sends any one host at most about one request a second
  • A crashed worker loses no URLs, and the crawl resumes where it stopped
  • Survives spider traps, malformed HTML, and slow servers

2 · Estimates

Drill This Phase First

Show the Answer and Estimates

Answer

About 400 a second

1B divided by 30 days is about 33M pages a day. 33M divided by 86,400 seconds is about 385, so plan for 400 a second.

Pages
1B pages a month: about 33M a day, or about 400 a second
Storage
About 500 KB a page: 500 TB a month, and about 30 PB over five years in object storage
Bandwidth
400 pages a second at 500 KB is about 200 MB a second, roughly 1.6 Gbps inbound
Fetchers
Each fetch waits about 1 second on the network, so about 400 stay in flight. Async workers holding 100 connections each need 4 nodes, and 10 leave room for recrawls
Seen URLs
About 5B known URLs in a Bloom filter at 1% false positives: about 10 bits each, roughly 6 GB in memory
Politeness
At one request a second per host, a 1M page site takes about 12 days, so large hosts get a crawl budget of their own

3 · API

POST /seeds { urls[], priority } → 202 (adds URLs to the frontier)
GET /urls/:url_hash → { status, last_crawled_at, next_crawl_at, content_hash }
PUT /hosts/:host/policy { max_rps, paused } → 204 (overrides the default politeness limit)
Event: page.fetched { url, fetched_at, blob_key, content_hash } → indexer topic

Data Model

urls (Cassandra, partition = url_hash):
  url_hash (PK)     : bytes, 8 bytes of SHA-1(normalized url)
  url               : string
  host              : string
  status            : enum(pending|fetched|failed|blocked)
  last_crawled_at   : timestamp?
  next_crawl_at     : timestamp    from the page's change rate
  content_hash      : bytes?       SimHash of the last fetch

hosts (Redis, key = "host:{host}"):
  robots_rules      : string       cached 24h
  next_allowed_at   : timestamp    politeness gate
  crawl_delay_ms    : int

pages (object storage, key = date/url_hash):
  raw HTML, gzip

4 · High-Level Design

A URL leaves the frontier only when its host is due. The page is fetched, stored, and parsed; new links pass the Bloom filter back into the frontier, and pages whose fingerprint matches a stored one never reach the indexer.

5 · Deep Dives

The URL Frontier

Split the frontier into two layers. Front queues order URLs by priority, from link rank or how often the page changes. Back queues hold one host each, and a worker takes from a back queue only once that host's next allowed time has passed. Map hosts to workers with consistent hashing, so one worker owns each host and politeness needs no coordination.

Duplicate URLs and Duplicate Content

Normalize each URL first: lowercase the host, drop fragments, and strip tracking parameters. Then check the Bloom filter. A false positive skips a new page about 1% of the time, a cost a crawler can accept. Mirrors and boilerplate make many pages near-copies, so fingerprint each page with SimHash and drop any within a few bits of one already stored.

Traps and Failures

Cap URL depth and length per host to escape spider traps such as endless calendar pages. Time out slow servers after a few seconds and retry with backoff. The frontier is persisted, and a worker acknowledges a URL only after its page is stored, so a crashed fetcher's URLs return to the queue.

6 · Wrap Up

Close with the next risks: rendering JavaScript-heavy pages in a headless browser pool, freshness for news sites that change by the minute, and fetchers in several regions so distant hosts answer faster.

The Short Version

A URL frontier with per-host queues enforces politeness. Fetchers resolve DNS through a local cache and store raw HTML in object storage. A Bloom filter skips seen URLs, and a SimHash fingerprint skips duplicate content.

Building Blocks