Replace Nightly Batch Billing With Streaming Usage Metering

Usage is rated once a night by a Spark job that reads the day's API gateway logs from S3. Usage charges are $14M of monthly revenue, customers see their usage up to 30 hours late, and the job has missed its 6 a.m. deadline 11 times this year.

Support handles about 400 tickets a month from customers whose own request counts disagree with the usage page.

Each API gateway publishes a UsageRecorded event to the usage-events Kafka topic for every billable request. A Flink job aggregates the events per customer and meter into hourly totals and writes them to the usage_hourly table in the billing database.

Each event carries the request ID, the customer ID, the meter name, and a quantity.

Hourly windows close at the top of each hour, based on when an event arrives at Flink. Events that arrive after their window has closed are dropped and counted in a metric.

Gateways retry a publish until Kafka acknowledges it, and the Flink job restarts from its last checkpoint after a failure, so no usage is lost.

The streaming meter and the nightly job run side by side for two weeks. If their daily totals for the 20 largest customers are within 2% of each other, we turn off the nightly job.

Cutover happens on the 15th of the month, so the team is on hand for the first invoice run after the switch. Invoices are built from usage_hourly from that day on.

The usage page reads from usage_hourly, so customers see each hour's usage within 5 minutes of the hour closing. The page labels the current day's totals as preliminary.

Raw events stay in Kafka for 7 days. After that, usage_hourly is the record of what each customer used.

Customers see their usage within an hour, down from up to 30 hours today.