Distributed GPU Serving & Scheduling
Distributed inference should make latency, throughput, and isolation predictable for real traffic—not merely maximize GPU utilization.
Model request shapes, parallelism, batching, routing, and failure domains together to engineer service levels.
01
Problem definition
Optimizing average token throughput can let long requests displace short-request time to first token.
When scheduling and model parallelism are designed separately, failure and retry costs become unpredictable.
- Multiple models or tenants share a constrained GPU pool
- Time-to-first-token, inter-token latency, and throughput SLOs must coexist
- A single low-traffic model fits comfortably on one machine
- Real request distributions and SLOs are undefined
Distributed GPU Serving & Scheduling: system plate
System 1
Build a load model reproducing input/output length, concurrency, priority, and bursts.
System 2
Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
System 3
Configure cache-aware routing, admission control, and preemption by SLO class.
System 4
Inject node, link, and worker failures to measure draining, retry, and recovery.
- N1 N2context
- N2 N3decision
- N3 N4evidence
- The workflow begins with Build a load model reproducing input/output length, concurrency, priority, and bursts..
- It reaches an acceptance decision through p50/p95 first-token and inter-token latency.
03
Design method
Distributed inference should make latency, throughput, and isolation predictable for real traffic—not merely maximize GPU utilization.
- 01
Stage 1
Build a load model reproducing input/output length, concurrency, priority, and bursts.
- 02
Stage 2
Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
- 03
Stage 3
Configure cache-aware routing, admission control, and preemption by SLO class.
- 04
Stage 4
Inject node, link, and worker failures to measure draining, retry, and recovery.
04
Application scenarios
Multiple models or tenants share a constrained GPU pool
Optimizing average token throughput can let long requests displace short-request time to first token.
- APPROACH
- Build a load model reproducing input/output length, concurrency, priority, and bursts.
- BOUNDARY
- Synthetic peak throughput is not presented as production SLO performance.
Time-to-first-token, inter-token latency, and throughput SLOs must coexist
When scheduling and model parallelism are designed separately, failure and retry costs become unpredictable.
- APPROACH
- Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
- BOUNDARY
- GPU utilization alone cannot represent user latency or reliability.
05
Design choices
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| Multiple models or tenants share a constrained GPU pool | Build a load model reproducing input/output length, concurrency, priority, and bursts. | Synthetic peak throughput is not presented as production SLO performance. | p50/p95 first-token and inter-token latency |
| Time-to-first-token, inter-token latency, and throughput SLOs must coexist | Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape. | GPU utilization alone cannot represent user latency or reliability. | Goodput and queue time |
Distributed GPU Serving & Scheduling: system plate
- A single low-traffic model fits comfortably on one machine
Synthetic peak throughput is not presented as production SLO performance.
- SIGNAL
- p50/p95 first-token and inter-token latency
- MITIGATION
- Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
- Real request distributions and SLOs are undefined
GPU utilization alone cannot represent user latency or reliability.
- SIGNAL
- Goodput and queue time
- MITIGATION
- Configure cache-aware routing, admission control, and preemption by SLO class.
- The workflow begins with Build a load model reproducing input/output length, concurrency, priority, and bursts..
- It reaches an acceptance decision through p50/p95 first-token and inter-token latency.
07
Validation plan
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| p50/p95 first-token and inter-token latency | Build a load model reproducing input/output length, concurrency, priority, and bursts. | Repeated runs satisfy the acceptance threshold agreed during discovery | Synthetic peak throughput is not presented as production SLO performance. |
| Goodput and queue time | Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Isolation and recovery time | Configure cache-aware routing, admission control, and preemption by SLO class. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
08
Constraints and failure conditions
A single low-traffic model fits comfortably on one machine
Synthetic peak throughput is not presented as production SLO performance.
Real request distributions and SLOs are undefined
GPU utilization alone cannot represent user latency or reliability.
09
Engagement model
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
p50/p95 first-token and inter-token latency - 02
Design
PattyCompare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
ClientConfirm owners and acceptance criteria.
Goodput and queue time - 03
Validation
PattyConfigure cache-aware routing, admission control, and preemption by SLO class.
ClientMake the production-transition or stop decision.
Isolation and recovery time
10
Durable deliverables
- Distributed GPU Serving & Scheduling decision record
- Distributed inference should make latency, throughput, and isolation predictable for real traffic—not merely maximize GPU utilization.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- p50/p95 first-token and inter-token latency · Goodput and queue time · Isolation and recovery timeJointly maintained
- Operations and recovery runbook
- Synthetic peak throughput is not presented as production SLO performance. · GPU utilization alone cannot represent user latency or reliability.Operating-team owned
11
Terminology
- Distributed GPU Serving & Scheduling
- Model request shapes, parallelism, batching, routing, and failure domains together to engineer service levels.
- Acceptance criterion
- p50/p95 first-token and inter-token latency
- Operating boundary
- Synthetic peak throughput is not presented as production SLO performance.
REFERENCES
References and primary material
- vLLM documentation
Primary material for the method and terminology.
- Ray Serve LLM
Primary material for the method and terminology.