Design a Shared ML Serving Platform
Fifteen teams each built their own serving stack, so a shared platform has to win on latency, correctness, and cost at once. A Principal answer fixes training and serving skew at the definition, budgets every millisecond, and turns a $2M monthly serving bill into a utilization number someone owns.
1 · Requirements
The Prompt
Design a shared feature and model serving platform for 15 product teams that each run their own pipelines today. Ranking calls need predictions in under 50 ms at p99, features drift between training and serving, and GPU spend is $3M a month, $2M of it on serving. Cover online and offline feature consistency, latency budgets, and GPU cost.
Functional
- Define a feature once and serve it online and offline from that definition
- Build training sets with point-in-time correct joins
- Register, shadow, canary, and roll back models
- Serve predictions from shared GPU and CPU pools with per-team quotas
- Report GPU hours and cost per team and per model
Non-Functional
- Ranking predictions return in under 50 ms at p99
- Served and training feature values match on at least 99.9% of sampled requests
- Peak GPU utilization reaches 60%, cutting serving spend to about $730,000 a month
- A failed model or feature falls back within the request deadline
2 · Estimates
Drill This Phase First
Show the Answer and Estimates
Answer
About 400 GPUs
240,000 times 50 is 12M scorings a second, and 12M divided by 50,000 is 240 GPUs of work. At 60% utilization that is 400 GPUs, against about 1,100 running today.
- Traffic
- 240,000 ranking requests a second at peak, 50 candidates each: 12M scorings a second across about 120 models
- GPU Work
- One batched GPU scores about 50,000 candidates a second, so peak load is 240 GPUs of work
- Today
- About 1,100 serving GPUs at $2.50 an hour cost about $2M a month and run at about 22% peak utilization
- Target
- 60% peak utilization needs 400 GPUs, about $730,000 a month, saving about $1.27M a month
- Online Store
- 300M entities with 200 features at 8 bytes: about 480 GB, about 1.5 TB with three replicas, in memory
- Feature Reads
- 240,000 multi-gets a second, each one user and 50 items, with the 10M hot items cached in process
- Latency Budget
- 50 ms at p99: 5 for routing, 10 for features, 20 for inference, 5 for post-processing, and 10 of headroom
- Training Data
- About 2B labeled events a day feed point-in-time joins in the lakehouse
3 · API
POST /v1/features { name, entity, source, transform, ttl, owner_team } → { feature_id, version }
POST /v1/features/online:get { entity_keys[], feature_names[] } → { values[][], stale[] }
POST /v1/training-sets { features[], labels_table, as_of_column } → { job_id } (point-in-time join)
POST /v1/models { name, artifact_uri, format, owner_team, latency_slo_ms } → { model_id, version }
POST /v1/predict/:model { request_id, user_id, candidate_ids[], deadline_ms } → { scores[], model_version, fallback_used }
POST /v1/models/:id/rollouts { version, mode: shadow | canary | full, traffic_pct } → { rollout_id }Data Model
feature_defs (SQL, registry):
feature_id (PK) : string user.ctr_7d
entity : enum(user|item|session)
transform : string one definition, both stores
version : int
owner_team : string
ttl_seconds : int
online_features (KV, key = entity_type:entity_id):
values : map<feature_id, bytes>
updated_at : timestamp
served_feature_log (lakehouse, partition = date, model):
request_id : string
model_version : string
features : map<feature_id, value> as served
score : float
models (SQL):
model_id (PK) : string
version : int
owner_team : string
hardware : enum(gpu|cpu)
gpu_hours_month : float chargeback4 · High-Level Design
5 · Deep Dives
One Definition, Two Stores
Skew comes from writing the same feature twice, once in a training notebook and once in a serving service. The registry holds one transform per feature, and the platform compiles it to batch and streaming jobs that write both stores. The gateway also logs the exact features it served, so new models train on served values, and a monitor compares a 1% sample against offline recomputation to catch drift.
Spending the 50 ms Budget
Give each hop a number: 5 ms for routing, 10 ms for one batched feature multi-get, 20 ms for inference, 5 ms for post-processing, and 10 ms of headroom. Features for the 10M hot items sit in an in-process cache, so the multi-get mostly fetches user features. The gateway passes a deadline down; when time runs short, it returns scores from a smaller CPU model or a cached ranking and records the fallback.
GPU Cost as a Platform Metric
Today 1,100 GPUs run at about 22% peak utilization because each team sizes for its own peak. A shared pool with multi-model servers and dynamic batching reaches 60%, which is 400 GPUs and about $1.27M a month saved. Move small and tree models to CPU, quantize where accuracy holds, and buy reserved capacity for the baseline with on-demand for peaks. Chargeback per 1,000 predictions shows every team the cost of its model.
Adoption, Ownership, and Standards
Start with the two teams that spend the most on GPUs and report the worst skew; their savings fund the rest of the rollout. The platform team owns the registry, both stores, the gateway, and the pools. Product teams own their features, models, and quality metrics. Standardize the feature definition API, entity keys, the model artifact format, and the prediction contract, since those are the one-way doors. Leave training frameworks open, and adopt an open-source inference server instead of building one.
6 · Wrap Up
The Short Version
Define each feature once, compute both stores from that definition, and log served features to catch skew. Split the 50 ms budget per hop, and cut GPU cost with pooled, batched serving, a utilization target, and per-team chargeback.