Design a Disaster Recovery Program
Disaster recovery needs an architecture and a program that exercises it: recovery time objectives (RTO) and recovery point objectives (RPO) only hold once restores and evacuations have actually run. A Principal answer prices each tier, sequences the work over years, and makes failure practice routine.
1 · Requirements
The Prompt
Design disaster recovery for a company running 900 services across two cloud regions. No service has a stated RTO or RPO, backups have never been restored at scale, and nobody has tested a region failover. Cover recovery tiers by service, region evacuation, backup and restore testing, and game days.
Functional
- Assign every service a tier with an RTO, an RPO, and a named owner
- Evacuate a region for tier 0 and 1 services from one controller
- Back up all data to an immutable vault in a separate account
- Restore every backup set on a schedule and record the measured time
- Run scheduled and unannounced game days, with findings tracked to closure
Non-Functional
- Tier 0 recovers within 15 minutes and loses under 1 minute of data
- Evacuation uses capacity already running in the healthy region
- Every backup set is restored at least once a quarter
- The program adds no more than about 15% to compute spend
2 · Estimates
Drill This Phase First
Show the Answer and Estimates
Answer
Yes, in about 11 hours
40 TB is 40,000 GB, and at 1 GB a second that is 40,000 seconds, about 11 hours. That leaves about 13 hours for validation and cutover, and the margin shrinks as the database grows.
- Tiers
- 40 services at tier 0, 160 at tier 1, 300 at tier 2, and 400 at tier 3, with RTOs of 15 minutes, 1 hour, 24 hours, and 7 days
- Traffic
- 600,000 requests a second at peak, and each region can carry all of it for tiers 0 and 1
- Headroom Cost
- Tiers 0 and 1 are $2.8M of an $8M monthly compute bill. Running at 50% utilization instead of 70% adds about $1.1M a month
- Replication
- 600 TB of tier 0 and 1 data replicates asynchronously with p99 lag under 30 seconds, inside the 1 minute RPO
- Transfer
- About 25 TB a day crosses regions at $0.02 a GB: about $15,000 a month
- Restore Time
- The largest tier 2 database, 40 TB at 1 GB a second, restores in about 11 hours against a 24 hour RTO
- Restore Tests
- About 1,200 backup sets restored every quarter: 13 to 14 automated restores a day
3 · API
PUT /v1/services/:id/tier { tier: 0..3, rto_minutes, rpo_minutes, owner_signoff } → 204
GET /v1/services/:id/dependencies → { deps[], violations[] } (a tier 0 service calling tier 2 is a violation)
POST /v1/evacuations { from_region, scope: tier0 | tier0-1 | all, mode: drill | real } → { evacuation_id }
GET /v1/evacuations/:id → { step, traffic_shifted_pct, replicas_promoted, elapsed_seconds }
POST /v1/restores { backup_set_id, target: scratch } → { restore_id }
GET /v1/restores/:id → { status, duration_seconds, rto_met, checksums_ok }Data Model
services (SQL, service catalog):
service_id (PK) : string
owner_team : string
tier : int 0 to 3
rto_minutes : int
rpo_minutes : int
last_evacuated : timestamp?
dependencies (SQL):
service_id : string
depends_on : string
kind : enum(sync|async)
backup_sets (SQL):
backup_set_id(PK): uuid
service_id : string
size_tb : float
last_restore_at : timestamp?
last_restore_min : int? measured, compared to the RTO
game_days (SQL):
game_day_id (PK) : uuid
scenario : enum(az_loss|region_evac|restore|dependency_kill)
announced : bool
findings : list<string>4 · High-Level Design
5 · Deep Dives
Tiers Are a Contract
Owners pick a tier and sign its RTO and RPO, and the catalog charges each tier's cost back to the owning org, which keeps every team from claiming tier 0. The dependency checker flags any service that calls a less critical tier synchronously; a first run typically finds login depending on a tier 2 service such as feature flags. Fix those before any failover test, or the test only proves the gap.
Region Evacuation
Tiers 0 and 1 run active-active, each region sized to carry the full 600,000 requests a second, so evacuation needs no new capacity from a region that may be failing. The controller shifts traffic in steps, fences the old primaries before promoting replicas, and accepts up to the RPO in lost writes, which are reconciled from the outbox after recovery. Tier 2 starts its pilot light in the healthy region and restores within its 24 hour RTO.
Backups That Restore
Backups land in an immutable vault in a separate account, so ransomware or a compromised admin cannot delete them. The verifier restores about 14 backup sets a day into scratch accounts, checks checksums and row counts, and records the measured time against each RTO. A failed or slow restore pages the owner like an outage. The 40 TB database at 11 hours passes today and gets flagged as it grows toward 24.
Sequencing the Program
Year one builds the catalog and tiers, fixes dependency violations, brings restore testing to every backup set, and ends with a planned read-only failover of tier 0. Year two runs full tier 0 and 1 evacuations, first in staging, then in production at low traffic, then quarterly. Year three makes evacuations unannounced. A small resilience team owns the controller, verifier, and calendar; service teams own their runbooks and tier targets. Buy backup tooling and traffic management, and build the catalog-driven controller.
6 · Wrap Up
The Short Version
Tier services by business impact with a priced RTO and RPO for each tier, and block any dependency on a less critical tier. A backup counts once an automated restore meets its RTO, and the program builds from restore tests to read failover to full, unannounced evacuations.