Design Doc · Checkout Team · In Review
Run Checkout Active-Active Across Two Regions
Press Enter to flag this paragraph.
Background
Checkout runs in us-east-1 with a cold standby in us-west-2. The us-east-1 outage on June 14 took checkout down for 3 hours and 40 minutes, and 2 hours of that went to restoring the standby from a database snapshot that was 9 hours old.
Goals
Checkout stays available through the loss of a full region, with zero downtime during regional failures. Shoppers in Europe also get lower checkout latency.
Proposal
Both us-east-1 and eu-west-1 serve live checkout traffic. Each region runs the full stack: the checkout API, the cart service, the payment orchestrator, and its own Aurora PostgreSQL cluster. Shoppers reach the nearer region through latency-based routing in Route 53.
Route 53 health checks probe each region's /health endpoint every 10 seconds. After 3 consecutive failures, DNS shifts that region's traffic to the other region, and shifts it back when the checks pass again.
Replication
Each region writes to its own cluster, and changes replicate to the other region through bidirectional logical replication with pglogical. Typical replication lag is under 1 second.
When the same row is updated in both regions before replication catches up, the write with the later timestamp wins. This applies to every checkout table, including inventory reservations and order status.
Customer profiles, addresses, and saved payment tokens replicate to both regions so either region can complete any shopper's checkout.
Cost
eu-west-1 runs at the same capacity as us-east-1 so either region can absorb all traffic on its own. The added infrastructure cost is covered by the reliability budget.
Rollout
eu-west-1 takes 5% of European traffic in April and all European traffic by the end of May. The us-west-2 standby is decommissioned in June.
Failover was validated in staging by scaling the us-east-1 checkout API to zero. A production failover drill will be scheduled after launch.
Non-Goals
Multi-region support for the merchant admin dashboard is out of scope for this project.
Open Questions
Whether to add a third region in ap-southeast-1 in 2027.