Design Doc CritiqueSelect any text to flag it
Staff

Design Doc · Platform Team · In Review

Move Notifications to a Shared Event Bus

Press Enter to flag this paragraph.

Problem

Each product team sends notifications through its own queue. We run 14 queues with 14 different retry policies, and notification pages to on-call rose from 31 in Q1 to 64 in Q3.

Proposal

Replace the per-team queues with one shared Kafka cluster. Teams publish NotificationRequested events to a single notifications topic with 48 partitions, using a schema the platform team maintains in the schema registry.

A central dispatcher service consumes the topic and fans out to the email, push, and SMS providers. It runs on the shared platform cluster. Ownership will move to whichever team ends up sending the most traffic.

Events are keyed by team ID, so each team's notifications are processed in the order they were sent.

Delivery

The dispatcher guarantees exactly once delivery to users. Product teams can rely on this and skip their own deduplication.

When a provider call fails, the dispatcher retries the event every 5 seconds until it succeeds.

Rollout

All teams switch over in one release on March 1. The old queues are deleted the following week.

Non-Goals

Per-team rate limits and priority lanes are out of scope for v1. Teams that need priority delivery can request it later.

Success

Notifications become simpler to operate and easier for teams to adopt.

Open Questions

None at this time.