Sweaty Imposter

Design an Ad Click Aggregator

System DesignWrite HeavyStreamingSenior3 min readUpdated

Advertisers are billed on these counts, so they have to be exact, and dashboards want them within a minute. The design counts clicks in a stream for speed and recounts the raw log in a nightly batch for billing.

1 · Requirements

The Prompt

Count 1B ad clicks a day per ad per minute for dashboards and billing. Counts must be exact for billing, and queries return in under a second.

Functional

  • Record each click and redirect the user to the advertiser
  • Count clicks per ad per minute
  • Query counts over a time range and the top ads in the last minute
  • Produce daily counts for billing

Non-Functional

  • Dashboards are at most a minute or two behind
  • Billing counts are exact: no click lost or counted twice
  • Queries return in under a second
  • Raw clicks are kept for a year for audits

2 · Estimates

Drill This Phase First

Show the Answer and Estimates

Answer

About 60,000 a second

1B divided by 86,400 seconds is about 11,600 a second on average. 5x that is about 58,000, so plan for 60,000.

Clicks
1B clicks a day: about 11,600 a second on average and 60,000 at a 5x peak
Raw Log
About 200 bytes a click: 200 GB a day and about 73 TB a year in object storage
Kafka
60,000 events a second at 200 bytes is 12 MB a second; 64 partitions keyed by ad_id keep each one light
Aggregates
About 200,000 ads get a click in a given minute: 288M rows a day at 50 bytes, roughly 15 GB
Dedup
One hour of click IDs at about 12,000 a second is about 43M IDs, under 1 GB of stream state
Workers
Each Flink worker aggregates about 10,000 events a second, so 6 cover peak and 12 cover a replay

3 · API

GET /c/:click_token → 302 to the advertiser URL (logs the click first)
GET /ads/:id/clicks?from=&to=&granularity=minute → { buckets[{ minute, clicks }] }
GET /ads/top?window=1m&n=100&country= → { ads[{ ad_id, clicks }] }
GET /billing/:advertiser_id?day= → { ad_counts[], reconciled }

Data Model

click event (Kafka "clicks", key = ad_id):
  click_id          : uuid           from the signed click token
  ad_id             : bigint
  ts                : timestamp      event time, set at the click server
  user_hash         : bytes
  country           : string

ad_clicks_minute (ClickHouse or Cassandra, partition = ad_id):
  ad_id, minute (PK)
  clicks            : bigint
  source            : enum(stream|batch)   batch overwrites stream

raw_clicks (object storage, Parquet, by hour):
  every event, kept 1 year for audits and replays

4 · High-Level Design

The stream path puts counts on dashboards within a minute. The batch path recounts every raw click overnight and overwrites the stream's numbers, so billing reads a count that was checked twice.

5 · Deep Dives

Event Time and Late Clicks

Bucket each click by when it happened, as stamped by the click server. Clicks arrive late from slow networks and retries, so a window closes only when the watermark passes its end plus a grace period, such as one minute. Anything later goes to a side output that the nightly batch counts.

Exact Counts

Every click carries a unique click_id from a signed token, and the aggregator drops IDs it saw in the last hour. Flink checkpoints its Kafka offsets and window state together, and the sink upserts each (ad_id, minute) row, so a replay after a crash writes the same totals again.

Stream Plus Batch

The stream gives dashboards numbers within a minute. Billing needs an audited record, so a nightly job recounts the day's raw clicks from object storage, overwrites the stream's rows, and flags any ad where the two differ by more than 0.1%. This is the lambda shape; a kappa design replays Kafka through the same streaming job.

6 · Wrap Up

Close with the next risks: click fraud from bots and click farms, a viral ad that overloads one Kafka partition, and retention rules for the raw logs.

The Short Version

Click servers log each click to Kafka, then redirect. A stream processor such as Flink counts per ad per minute with event-time windows and watermarks. A nightly batch recount over raw clicks in object storage reconciles the numbers used for billing.

Building Blocks