Design Doc · Platform Team · In Review
Move Notifications to a Shared Event Bus
Press Enter to flag this paragraph.
Problem
Each product team sends notifications through its own queue. We run 14 queues with 14 different retry policies, and notification pages to on-call rose from 31 in Q1 to 64 in Q3.
Proposal
Replace the per-team queues with one shared Kafka cluster. Teams publish NotificationRequested events to a single notifications topic with 48 partitions, using a schema the platform team maintains in the schema registry.
A central dispatcher service consumes the topic and fans out to the email, push, and SMS providers. It runs on the shared platform cluster. Ownership will move to whichever team ends up sending the most traffic.
Events are keyed by team ID, so each team's notifications are processed in the order they were sent.
Delivery
The dispatcher guarantees exactly once delivery to users. Product teams can rely on this and skip their own deduplication.
When a provider call fails, the dispatcher retries the event every 5 seconds until it succeeds.
Rollout
All teams switch over in one release on March 1. The old queues are deleted the following week.
Non-Goals
Per-team rate limits and priority lanes are out of scope for v1. Teams that need priority delivery can request it later.
Success
Notifications become simpler to operate and easier for teams to adopt.
Open Questions
None at this time.