Design an Internal Deploy Platform
Every team's release depends on this platform, so its failures and its rules reach the whole company. The interview turns on who owns what, how a bad change is caught at a fraction of a percent of traffic, and how rollback keeps working when the platform itself is down.
1 · Requirements
The Prompt
Design the deployment platform that 400 services and 60 teams ship through across 3 regions, with progressive rollouts, automatic rollback on bad metrics, and policy gates such as change freezes and required reviews. Today each team deploys with its own scripts on shared CI.
Functional
- Ship a signed artifact through canary, regional waves, and full rollout
- Compare canary metrics against a baseline and roll back automatically
- Enforce policy gates: freezes, required reviews, and service tier rules
- Let teams pause, promote, or roll back, with a break-glass path for emergencies
- Record every deploy, decision, and override for audit
Non-Functional
- Rollback works when CI, the control plane, or the metrics backend is down
- A bad change reaches at most 0.33% of traffic before analysis can stop it
- Detect a bad canary within 5 minutes and restore within 2 more
- About 120 concurrent rollouts at peak, with no team's deploy blocking another's
2 · Estimates
Drill This Phase First
Show the Answer and Estimates
Answer
About 120
80% of 1,200 is 960 deploys over 8 hours, or 120 an hour. Each lasts an hour, so about 120 are in flight at once.
- Deploys
- 400 services at about 3 deploys a workday: 1,200 a day, about 120 an hour in the busy 8 hours
- In Flight
- About an hour per rollout with bake times, so about 120 rollouts run at once at peak
- Stages
- Canary, region 1 at 25% and 100%, region 2, and region 3: 6,000 stage transitions a day
- Analysis
- 120 rollouts, 10 SLIs each, canary and baseline every 30 seconds: about 80 metric queries a second
- Fleet
- About 50 instances per service: 20,000 instances across 3 regions
- Blast Radius
- A canary at 1% of one region is about 0.33% of global traffic
- Artifacts
- 1,200 images a day at about 300 MB, mostly shared layers, pulled from a registry mirror in each region
3 · API
POST /v1/deploys { service, artifact_digest, change_ref, strategy? } → { deploy_id, plan[] }
GET /v1/deploys/:id → { stage, status, analysis[], gates[] }
POST /v1/deploys/:id/rollback { reason } → 202 (always allowed, no gates)
POST /v1/deploys/:id/promote | /pause → 202
PUT /v1/services/:name/deploy-spec { stages[], slis[], bake_minutes, owners[] } → 204 (synced from the service repo)
POST /v1/freezes { scope, starts_at, ends_at, exceptions[] } → { freeze_id }
POST /v1/break-glass { deploy_id, gate, ticket } → 202 (audited, pages the owning team)Data Model
deploys (SQL):
deploy_id (PK) : uuid
service : string
artifact_digest : string sha256, signed by CI
change_ref : string PR or ticket
requested_by : string
status : enum(planned|running|paused|rolled_back|done)
created_at : timestamp
stages (SQL):
deploy_id, seq : uuid, int composite PK
region : string
target_pct : int
bake_until : timestamp
verdict : enum(pass|fail|inconclusive)?
service_specs (SQL, synced from repos):
service (PK) : string
owner_team : string
tier : enum(1|2|3)
stages, slis : json
policy_version : string
service_locks (SQL):
service (PK) : string
deploy_id : uuid
lease_expires_at : timestamp
deploy_events (append-only):
event_id, deploy_id, type, actor, payload, ts4 · High-Level Design
5 · Deep Dives
Rollouts as Durable State Machines
Each deploy becomes a plan of stages stored in a database, and the orchestrator advances one stage at a time after its bake time and analysis verdict. Workers are stateless and claim rollouts with a lease, so a crashed worker's rollouts resume elsewhere. One lock per service serializes its deploys; a newer artifact queues behind the current rollout or supersedes it if the owner chooses. Regional executors pull desired state and apply it idempotently, so a lost message delays a stage and a repeated one changes nothing.
Automatic Rollback Without False Alarms
Canary analysis compares the canary against a baseline started from the old version at the same time and size, so time-of-day traffic swings cancel out. Each service declares its SLIs, with error rate, latency, and saturation as defaults. A statistical test needs a minimum sample, so a low-traffic service gets a longer bake. An inconclusive result pauses the rollout and asks the owner. A failed result rolls back without asking, because a rollback costs minutes and a bad release can cost hours.
Policy Gates and Team Boundaries
The platform owns the rollout mechanics, the policy engine, and the audit log. Teams own their deploy spec, their SLIs, and their bake times, inside limits set by service tier. Policies are code, versioned and reviewed: a freeze blocks tier 1 promotions, and a payments service needs two reviewers. Break-glass lets an on-call engineer override a gate with a ticket, records the override, and pages the owning team. Rollback is never gated.
When the Platform Itself Fails
Rollback needs the fewest dependencies of any path. Regional executors cache the last two good versions of every service and can restore one with only the cluster API available, even when the control plane, CI, and the metrics backend are all down. The platform ships itself through a separate, minimal pipeline that runs its previous stable release. Executors run per region, so a regional outage stalls only that region's rollouts.
Moving 60 Teams Off Their Scripts
Start by wrapping existing scripts as a stage type, so a team gains audit, freezes, and a rollback button without rewriting anything. Onboard tier 3 services first, publish lead time and change failure rate per team, and move tier 1 services once the platform has a quarter of clean history. When 90% of deploys run through the platform, cluster admission starts rejecting artifacts the platform did not sign, with a dated exception list for the rest.
6 · Wrap Up
The Short Version
Model each rollout as a durable state machine that advances on canary analysis and rolls back by default, with policy evaluated as code at every stage. Keep the rollback path independent of the control plane, and win adoption team by team before the clusters reject unsigned artifacts.