Design a Metrics and Alerting Pipeline
Every team debugs and gets paged through this system, so it has to stay up during the incidents it exists to catch. The interview turns on cardinality, tenant isolation on both the write and read paths, and alert routing that reaches the right person.
1 · Requirements
The Prompt
Design company-wide metrics and alerting for 300 teams: about 50M active time series scraped every 10 seconds, 13 months of retention, and pages that reach the right on-call within a minute. One team's cardinality explosion must never degrade another team's dashboards or alerts.
Functional
- Ingest metrics from every host and service, labeled by team, service, and environment
- Query recent and historical data for dashboards and ad hoc debugging
- Evaluate alert rules and route each alert to the owning team's on-call
- Show each team its series count, top labels, and quota
Non-Functional
- About 5M samples a second, with every team held to its own limits
- Alerts fire within 60 seconds of the condition
- Alerting keeps working when dashboards are overloaded
- 13 months of retention at a cost each team can see
2 · Estimates
Drill This Phase First
Show the Answer and Estimates
Answer
About 650 GB a day
50M divided by 10 is 5M samples a second. Times 86,400 is 432B samples a day, and at 1.5 bytes that is about 650 GB, before replication.
- Samples
- 50M series every 10 seconds: 5M samples a second, 15M writes a second with 3 replicas
- Ingesters
- About 250,000 samples a second each, so 60 ingesters on a hash ring
- Raw Storage
- 432B samples a day at about 1.5 bytes: roughly 650 GB a day and 9.7 TB for 15 days
- Rollups
- 5-minute points at about 6 bytes: 14.4B points a day, 86 GB a day, and 34 TB for 13 months
- Object Storage
- About 44 TB in total once the compactor removes the replica copies
- Index
- About 1 KB per active series: 150 GB across 60 ingesters with 3 replicas, about 2.5 GB each
- Rules
- 40,000 alert rules every 30 seconds: about 1,300 evaluations a second
- Quotas
- 100,000 series per team by default; a user_id label over 2M users would add 4% of the whole fleet
3 · API
POST /api/v1/push (protobuf, snappy; X-Tenant header) → 200 | 429 { limit, current, max }
GET /api/v1/query_range?query=…&start=…&end=…&step=… → { series[] } (capped series and samples per tenant)
PUT /api/v1/rules/:tenant/:group { rules[] } → 202 (linted and dry-run evaluated)
PUT /api/v1/limits/:tenant { max_series, max_series_per_metric, ingest_rate, query_concurrency } → 204 (platform admins)
GET /api/v1/cardinality/:tenant?top=20 → { metrics[{ name, series, top_labels[] }] }
PUT /api/v1/routes/:team { matchers, receivers, escalation_policy } → 204Data Model
series (ingester memory + block index):
series_id : hash(tenant, sorted labels)
labels : map<string, string> required: team, service, env
chunks : compressed (ts, value), ~1.5 B per sample
blocks (object storage):
{tenant}/{block_ulid}/index, chunks/, meta.json
resolution : enum(raw|5m)
min_ts, max_ts : timestamp
tenant_limits (SQL, cached in every distributor):
tenant (PK) : string team id
max_active_series : int default 100K
max_series_per_metric : int default 20K
ingest_samples_per_sec : int
query_max_series : int
query_concurrency : int
alert_state (rulers, replicated in pairs):
rule_id, fingerprint : string
state : enum(pending|firing|resolved)
active_since : timestamp
routes (SQL):
team, matchers, receiver, escalation_policy, inhibit_rules4 · High-Level Design
5 · Deep Dives
Cardinality Limits at the Front Door
Distributors check every incoming series against its tenant's limits before it reaches an ingester: total active series, series per metric name, and samples per second. New series over a limit are dropped and counted while existing series keep flowing, so a bad label stops growing and the team keeps its other signals. A HyperLogLog sketch per metric estimates cardinality cheaply, and the cardinality API lists the top labels so the team can find the user_id that caused it.
Tenant Isolation on the Read Path
A query frontend splits long ranges by day, caches the results, and queues queries per tenant with fair scheduling, so one team's 30-day dashboard cannot starve the others. Each tenant has caps on series fetched, samples scanned, and concurrent queries. Rulers run on their own queriers and read recent data straight from the ingesters, so a dashboard surge during an incident leaves alert evaluation on schedule.
Alert Routing That Reaches the Owner
Every series carries team and service labels, enforced at ingest, and routes match on them to reach the owning on-call schedule. Rulers run in pairs for availability, and the router dedupes their matching alerts by fingerprint. Grouping folds 200 pod alerts into one page, and inhibition silences symptoms while their cause is firing. A separate meta-monitoring stack sends a heartbeat through an external paging provider every minute; when it stops, the platform on-call is paged that the pipeline itself is down.
Migrating From 120 Prometheus Servers
Teams run about 120 Prometheus servers today, many near their memory limits. Turn on remote write from each one first, so data lands in the new system with no instrumentation changes. Import their rules and run them in shadow mode, logging what would fire, and compare firing sets for two weeks. Then cut paging over team by team, and retire each server once its dashboards point at the new backend.
6 · Wrap Up
The Short Version
Shard series across replicated ingesters on a hash ring, flush compressed blocks to object storage, and enforce per-tenant series and query limits at the front door. Run alert evaluation on its own capacity, route by ownership labels, and watch the whole pipeline from an independent stack.