Compare Shadow Reads Before a Cutover
Accounts are moving to a new datastore, and the decision to cut over will rest on a shadow comparison running in production. The README has the rules and what each one protects. Build shadow.py with the assistant, and judge each suggestion by whether the number it produces can be trusted and what it risks on the login path.
shadow.py1 of 3
Files
shadow.py
Python · Editableclass ShadowReader:
"""Serves reads from the current store, and compares a sample with the new store before cutover."""
IGNORED = {"updated_at"} # the new store keeps milliseconds, the old one seconds
def __init__(self, current, candidate, sample_rate, rng, log):
self.current = current
self.candidate = candidate
self.sample_rate = sample_rate
self.rng = rng
self.log = log # log(event, **fields) ships to the logging vendor
self.stats = {"compared": 0, "mismatched": 0, "errors": 0}
def get(self, key):
record = self.current.get(key)
return record
def _shadow(self, key, record):
raise NotImplementedError
def _compare(self, key, record, other):
self.stats["compared"] += 1
raise NotImplementedError
def mismatch_rate(self):
"""The cutover gate. A read the candidate failed counts against it."""
raise NotImplementedError
from shadow import ShadowReader
ANA = {"id": "u1", "email": "ana@example.com", "plan": "pro", "updated_at": 1700000000}
class Store:
def __init__(self, records, fail=False):
self.records = records
self.fail = fail
def get(self, key):
if self.fail:
raise TimeoutError("candidate timed out")
return self.records.get(key)
class FixedRandom:
def __init__(self, value):
self.value = value
def random(self):
return self.value
def make(candidate=None, fail=False, roll=0.0, rate=1.0):
logged = []
reader = ShadowReader(
Store({"u1": ANA}),
Store({"u1": ANA} if candidate is None else candidate, fail),
rate,
FixedRandom(roll),
lambda event, **fields: logged.append((event, fields)),
)
return reader, logged
def test_reads_come_from_the_current_store():
reader, _ = make({"u1": {**ANA, "plan": "free"}})
assert reader.get("u1") == ANA
def test_sampled_reads_are_compared():
reader, logged = make({"u1": {**ANA, "updated_at": 1700000000.123}})
reader.get("u1")
assert (reader.stats["compared"], reader.stats["mismatched"], reader.stats["errors"]) == (1, 0, 0)
assert logged == []
def test_unsampled_reads_skip_the_candidate():
reader, _ = make(fail=True, roll=0.5, rate=0.01)
assert reader.get("u1") == ANA
assert reader.stats["compared"] == 0
def test_mismatch_is_logged_by_field():
reader, logged = make({"u1": {**ANA, "email": "ana@old.example.com"}})
reader.get("u1")
assert reader.stats["mismatched"] == 1
assert [fields.get("fields") for _, fields in logged] == [["email"]]
def test_logs_never_contain_customer_data():
reader, logged = make({"u1": {**ANA, "email": "ana@old.example.com"}})
reader.get("u1")
assert "ana@" not in repr(logged), "customer data in the logs"
def test_candidate_failure_never_fails_the_read():
reader, logged = make(fail=True)
assert reader.get("u1") == ANA
assert reader.stats["errors"] == 1
assert logged == [("shadow_error", {"key": "u1", "error": "TimeoutError"})]
def test_rate_counts_failures_against_the_candidate():
reader, _ = make({"u1": {**ANA, "plan": "free"}})
reader.get("u1")
reader.candidate.fail = True
reader.get("u1")
reader.candidate.fail = False
reader.candidate.records = {"u1": ANA}
reader.get("u1")
reader.get("u1")
assert reader.mismatch_rate() == 0.5
def test_rate_is_undefined_before_any_comparison():
reader, _ = make()
assert reader.mismatch_rate() is None
# Shadow Reads
Accounts are moving from Postgres to a new document store. Before any read is served from the new store, ShadowReader (shadow.py) keeps serving every read from Postgres and compares a sample against the new store.
## Rules
- Reads always come from the current store. Nothing about the candidate store, slow, failing, or wrong, may change what the caller gets or make the read fail.
- sample_rate of reads are compared. updated_at is ignored, since the new store keeps milliseconds and the old one keeps seconds.
- A mismatch is counted and logged with the record's key and the names of the fields that differ.
- mismatch_rate is the cutover gate. A read the candidate failed counts as a mismatch, and with no comparisons yet the rate is None, so the gate can't pass on an empty sample.
## Operational Stakes
- The cutover decision rests on mismatch_rate staying under 0.1% for a week at a 1% sample. A rate that leaves out failures makes a flaky store look ready.
- Logs ship to a third-party vendor with a 13 month retention. Customer data in a log line is a reportable privacy incident.
- Accounts reads sit on the login path, so any exception here signs people out.
The tests in test_shadow.py use fake stores, a fixed random source, and a log that records its calls.
Test Output
No runs yet.