Roll Out Feature Flags by Percentage
Product ramps features through flags, and the API runs as many processes that restart on every deploy. The README has the rules and what goes wrong in production when assignment drifts. Build flags.py with the assistant, and judge each suggestion by how it behaves across the whole fleet.
flags.py1 of 3
Files
flags.py
Python · Editablefrom dataclasses import dataclass, field
BUCKETS = 10_000
@dataclass
class Flag:
name: str
percent: float = 0.0 # share of users, 0 to 100
allow: set = field(default_factory=set) # user ids always on
deny: set = field(default_factory=set) # user ids always off
killed: bool = False
def __post_init__(self):
pass
def bucket(flag_name, user_id):
"""A number from 0 to BUCKETS - 1, the same for a flag and user on every host and every deploy."""
raise NotImplementedError
def in_rollout(flag, user_id):
"""True when the user's bucket falls inside the flag's percentage."""
raise NotImplementedError
def is_enabled(flag, user_id):
return in_rollout(flag, user_id)
import os
import subprocess
import sys
import pytest
from flags import BUCKETS, Flag, bucket, is_enabled
USERS = [f"user-{n}" for n in range(100_000)]
def buckets_in_a_new_process(seed):
code = "from flags import bucket; print([bucket('checkout-v2', f'user-{n}') for n in range(50)])"
env = {**os.environ, "PYTHONHASHSEED": seed}
run = subprocess.run([sys.executable, "-c", code], env=env, capture_output=True, text=True)
assert run.returncode == 0, run.stderr.strip().splitlines()[-1]
return run.stdout
def test_bucket_is_in_range():
assert all(0 <= bucket("checkout-v2", u) < BUCKETS for u in USERS[:1000])
def test_buckets_match_across_processes():
first, second = buckets_in_a_new_process("1"), buckets_in_a_new_process("2")
assert first == second, "buckets differ between processes"
def test_zero_percent_enables_no_one():
flag = Flag("checkout-v2", percent=0)
assert not any(is_enabled(flag, u) for u in USERS)
def test_full_rollout_enables_everyone():
flag = Flag("checkout-v2", percent=100)
assert all(is_enabled(flag, u) for u in USERS)
def test_rollout_is_close_to_its_percent():
flag = Flag("checkout-v2", percent=10)
share = sum(is_enabled(flag, u) for u in USERS) / len(USERS)
assert 0.095 < share < 0.105
def test_ramping_up_only_adds_users():
early = {u for u in USERS[:20_000] if is_enabled(Flag("checkout-v2", percent=5), u)}
later = {u for u in USERS[:20_000] if is_enabled(Flag("checkout-v2", percent=25), u)}
assert early <= later
def test_deny_list_and_kill_switch_win():
flag = Flag("checkout-v2", percent=100, allow={"user-1"}, deny={"user-2"})
assert not is_enabled(flag, "user-2")
flag.killed = True
assert not is_enabled(flag, "user-1")
def test_allow_list_works_at_zero_percent():
flag = Flag("checkout-v2", percent=0, allow={"user-1"})
assert is_enabled(flag, "user-1")
def test_rejects_a_percent_outside_0_to_100():
for percent in [-1, 100.5]:
with pytest.raises(ValueError):
Flag("checkout-v2", percent=percent)
# Percentage Rollouts
Product ramps new features through flags: 1% of users, then 5%, 25%, and 100%. is_enabled (flags.py) decides for each request. The API runs as about 40 processes behind a load balancer, and they restart on every deploy.
## Rules
- A user's bucket depends only on the flag name and the user id. It must be the same on every process and after every restart, or users see a feature appear and disappear between requests, and experiment results stop meaning anything.
- percent of the buckets are in the rollout: 0 enables no one, and 100 enables everyone. Raising the percent only adds users.
- Each flag picks its own users, so the same 1% don't get every experiment first.
- The kill switch turns a flag off for everyone. The deny list beats the allow list, and both beat the percentage.
- percent is 0 to 100.
## Operational Stakes
- Rollout decisions are logged and joined to conversion data. If assignment changes between processes, both groups get a mix of users who saw the feature and users who didn't, and the analysis quietly reports no effect.
- The kill switch is the first response to an incident. Nothing may turn a killed flag back on for anyone.
The tests in test_flags.py run bucket in fresh processes to check that it is stable.
Test Output
No runs yet.