Design a Multi-Region Payments Ledger
A ledger is the system of record that every product team and the finance org depend on. The interview turns on where correctness is enforced, how a region fails without losing a cent, and how fifteen teams move off the database they write to today.
1 · Requirements
The Prompt
Design the double-entry ledger for a payments company running in two regions: 40M transactions a day, every transaction balanced to the cent, daily reconciliation against bank and processor files, and an immutable audit trail. Fifteen product teams write directly to a shared Postgres database today.
Functional
- Post multi-leg transactions through approved templates, each balanced per currency
- Place and release holds, and correct mistakes only with reversing entries
- Read any account's balance and entries as of any point in time
- Reconcile daily against processor and bank files, with a queue for every break
- Give auditors a complete, tamper-evident history of every change
Non-Functional
- A region failure loses no posted entry and recovers in under 5 minutes
- A retried request posts exactly once
- About 2,300 transactions a second at peak with p99 commit under 250 ms
- Seven years of history stays queryable for audit
2 · Estimates
Drill This Phase First
Show the Answer and Estimates
Answer
About 100 TB
40M times 4 is 160M postings a day. At 250 bytes that is 40 GB a day, about 14.6 TB a year, and about 102 TB over 7 years, before replicas.
- Transactions
- 40M a day, about 460 a second on average and 2,300 at a 5x holiday peak
- Postings
- About 4 entry lines per transaction: 160M postings a day and 9,300 a second at peak
- Storage
- About 250 bytes a posting: 40 GB a day, 14.6 TB a year, and about 100 TB for 7 years of retention
- Shards
- 32 shards, 16 homed in each region, at about 290 postings a second each at peak
- Commit
- About 70 ms between regions, so a synchronous commit lands near 80 ms at p50 against a 250 ms p99 budget
- Hot Accounts
- The fee account takes every capture, up to 2,300 writes a second; 64 sub-accounts bring that to about 36 each
- Reconciliation
- About 40M external lines a day from 6 processors and 12 banks, matched within 4 hours: roughly 2,800 matches a second
3 · API
POST /v1/transactions { idempotency_key, template, legs[{ account_id, direction, amount_minor, currency }], metadata } → { txn_id, status: posted }
POST /v1/transactions/:id/reverse { idempotency_key, reason } → { reversal_txn_id } (corrections are new entries; nothing is edited)
POST /v1/holds { idempotency_key, account_id, amount_minor, currency, expires_at } → { hold_id }
DELETE /v1/holds/:id → 204 (release, or capture through a transaction template)
GET /v1/accounts/:id/balance?as_of=ts → { posted, held, available, version }
GET /v1/accounts/:id/entries?cursor=… → { entries[], next_cursor }
GET /v1/recon/breaks?status=open → { breaks[] } (finance ops queue)Data Model
transactions (SQL, on the home shard, append-only):
txn_id (PK) : uuid v7
idempotency_key : string unique per (producer, key)
template : string e.g. card_capture.v3
effective_at : timestamp
reverses_txn_id : uuid?
prev_hash : bytes hash chain per shard
producer : string calling service
entries (SQL, same shard and transaction):
txn_id : uuid
account_id : uuid
direction : enum(debit|credit)
amount_minor : bigint integer minor units
currency : char(3)
CHECK : debits = credits per txn and currency
account_balances (SQL, updated in the same transaction):
account_id (PK) : uuid fee accounts split into 64 sub-accounts
posted_minor : bigint
held_minor : bigint
version : bigint optimistic concurrency
recon_breaks (SQL):
break_id (PK) : uuid
source : enum(bank|processor)
external_ref : string
ledger_txn_id : uuid?
delta_minor : bigint
status : enum(open|explained|resolved)
owner, due_at : string, timestamp4 · High-Level Design
5 · Deep Dives
Correctness Lives in the Ledger
Product teams call a posting API with a named template such as card_capture.v3, which the ledger expands into entry lines. The ledger checks that debits equal credits per currency, stores amounts as integer minor units, and writes the idempotency key, the entries, and the balance updates in one database transaction. A retry with the same key returns the original result. Hot accounts such as platform fees split into 64 sub-accounts, so no single row becomes the bottleneck.
Two Regions With Zero Data Loss
Home each account to one region by its legal entity. Each shard runs a leader and a synchronous replica in its home region plus a synchronous standby in the other region, and a commit waits for that standby. That costs one 70 ms round trip per commit and buys a failover that loses nothing. A lease held by a small consensus group with a witness in a third location decides which side may write, so a network split cannot produce two leaders. Transfers between accounts homed in different regions run as a saga of local, balanced transactions through an inter-region clearing account.
Reconciliation and the Audit Trail
Every minute, a job sums all entries per shard and currency and pages if the trial balance is anything other than zero. Daily, the reconciliation engine matches processor settlement files and bank statements against ledger entries by external reference, amount, and date. Each mismatch opens a break with an owner and an aging deadline, and finance ops clears it with an explained adjustment. Journal rows are append-only and hash chained per shard, and CDC streams them to write-once object storage that auditors can replay.
Migrating Off the Shared Database
Put the ledger API in front of the old tables first, so every team writes through one interface before any storage moves. Then shadow write each posting into the new ledger and compare balances every hour. Move product lines one at a time: post opening balances as a migration journal entry, run both systems in parallel until a month of reconciliation matches to the cent, and then make the old tables read-only. Revoke direct database credentials last, once the access logs show no writers.
6 · Wrap Up
The Short Version
Enforce balance, idempotency, and append-only history inside the ledger, and replicate every shard synchronously into the second region so a failover loses nothing. Reconcile continuously with an internal trial balance and daily against external files, and migrate teams through a reconciled parallel run.