Design a Global Config Rollout System
Config changes cause as many outages as code deploys, and they reach every host in seconds. The interview turns on validating and staging changes that propagate that fast, keeping hosts healthy when the control plane is gone, and a kill switch that works on the worst day.
1 · Requirements
The Prompt
Design a dynamic configuration system that pushes changes to 20,000 hosts across 4 regions within 10 seconds. It needs schema validation before anything ships, staged rollout with automatic revert, and a kill switch that works when the rest of the system is down. About 200 teams own 50,000 keys spread across three legacy flag systems.
Functional
- Teams own namespaces of typed keys with schemas, reviewers, and validators
- Stage every change from canary hosts to global, with automatic revert
- Engage a kill switch globally in seconds
- Show which revision every host is running
- Keep a full history of every change, its author, and its rollout
Non-Functional
- A released stage reaches its hosts in under 10 seconds
- Hosts keep serving on last-known-good config when the control plane is down
- A control plane restart never triggers a thundering herd
- No change reaches more than 1% of hosts before validation and canary health pass
2 · Estimates
Drill This Phase First
Show the Answer and Estimates
Answer
About 80 GB
20,000 times 4 MB is 80,000 MB, about 80 GB in one burst. Agents send their current revision and get only the diff, with reconnects jittered over 30 seconds.
- Hosts
- 20,000 across 4 regions; 20 distributors per region hold about 250 agent streams each
- Keys
- 50,000 keys at about 2 KB: 100 MB in total, and about 4 MB for a typical host's 2,000 keys
- Changes
- About 5,000 a day, roughly 10 a minute during working hours
- Fan-Out
- A 2 KB change to every host is 40 MB; at 10 a minute that is about 6.7 MB a second worst case
- Latency
- About 50 ms to commit, 150 ms across regions, 100 ms to push, and under a second to apply: about 1.3 seconds
- Reconnects
- Full snapshots would move 80 GB; deltas with 30 seconds of jitter mean about 670 small reconnects a second
- Canary
- 1% of the fleet, 200 hosts from one region, bakes for 5 minutes before the 10% stage
3 · API
POST /v1/changes { key, value, change_ref } → { change_id, validation[] } (schema, owner checks, and dry run)
POST /v1/rollouts { change_id, strategy: staged } → { rollout_id, stages[] }
POST /v1/rollouts/:id/halt | /revert → 202
POST /v1/kill-switches/:key/engage { reason } → { revision } (skips staging, signed, audited)
Agent stream: Watch { namespaces[], since_revision } → stream { revision, key, value | tombstone }
GET /v1/hosts/:id/config?namespace=… → { applied_revision, values } (what a host actually runs)Data Model
config_keys (Raft store):
key (PK) : string namespace/name, e.g. payments/retry_policy
owner_team : string
schema_ref : string JSON Schema or protobuf type
kind : enum(config|flag|kill_switch)
safe_default : bytes kill switches move only to this
config_versions (Raft store, append-only):
key : string
revision : bigint global, monotonic
value : bytes
author : string
change_ref : string
created_at : timestamp
rollouts (SQL):
rollout_id (PK) : uuid
key : string
from_rev, to_rev : bigint
stage : enum(canary|10pct|50pct|global|halted|reverted)
cohort : hash(host_id) range
started_at : timestamp
agent_status (distributor memory, reported every 30 s):
host_id, region, applied_revision, last_seen4 · High-Level Design
5 · Deep Dives
Validation and Ownership
Each namespace belongs to a team and lives in a repo with its schemas, so a change goes through review like code. The validator checks types and ranges against the schema, runs the owner's custom checks, and replays the new value through the consuming client library in a dry run to catch values that parse cleanly and still crash. A change that touches a tier 1 service needs a second reviewer, and freezes from the deploy platform apply to config too.
Staged Rollout With Automatic Revert
A release targets cohorts by hash(host_id), so the same 200 canary hosts see every change first and their health history stays comparable. The rollout controller watches the consuming services' error rates and crash loops on those hosts for 5 minutes, then widens to 10%, 50%, and global, region by region. A regression halts the rollout and publishes the previous revision to the affected hosts. Every revision is immutable, so a revert is one more release.
Static Stability and the Kill Switch
Agents write each applied revision to local disk, and processes read config through the agent, so a host keeps running on last-known-good values with the whole control plane down, and restarts with them too. Kill switches are pre-registered boolean keys with a safe default. Engaging one skips staging, which is safe because it can only turn a feature off. It travels the normal stream and a second path: a signed file in object storage that agents poll every 30 seconds while their stream is down.
Fan-Out and Reconnect Storms
The Raft cluster serves only the 80 distributors, so its load stays flat as the fleet grows. Distributors hold long-lived streams and send each agent only the keys it subscribes to. An agent reconnects with its current revision and receives the diff, with jittered backoff spread over 30 seconds, so a distributor restart moves kilobytes per host where full snapshots would move 80 GB. Distributors also report each host's applied revision, so the console shows exactly which hosts lag.
Consolidating Three Flag Systems
Today config arrives three ways: baked into deploy artifacts, a homegrown flag service polled every 60 seconds, and a vendor SDK. Ship the agent with adapters that serve the old client APIs from the new store, then import each system's keys and compare what both return on every read for two weeks. Move namespaces team by team, freeze writes to the old system as each one moves, and remove a legacy client once its read counts reach zero.
6 · Wrap Up
The Short Version
Store versioned config in a small consensus cluster and fan it out through regional distributors to an agent on every host that keeps a last-known-good copy on disk. Treat every change like a deploy, with validation and staged rollout, and give kill switches an independent path that only moves toward a safe default.