Distributed GPU Serving & Scheduling
Distributed inference should make latency, throughput, and isolation predictable for real traffic—not merely maximize GPU utilization.
Model request shapes, parallelism, batching, routing, and failure domains together to engineer service levels.
§ 01
The operating conditions that justify Distributed GPU Serving & Scheduling
Optimizing average token throughput can let long requests displace short-request time to first token.
When scheduling and model parallelism are designed separately, failure and retry costs become unpredictable.
- Multiple models or tenants share a constrained GPU pool
- Time-to-first-token, inter-token latency, and throughput SLOs must coexist
- A single low-traffic model fits comfortably on one machine
- Real request distributions and SLOs are undefined
Distributed GPU Serving & Scheduling: system plate
System 1
Build a load model reproducing input/output length, concurrency, priority, and bursts.
System 2
Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
System 3
Configure cache-aware routing, admission control, and preemption by SLO class.
System 4
Inject node, link, and worker failures to measure draining, retry, and recovery.
- N1 N2context
- N2 N3decision
- N3 N4evidence
- The workflow begins with Build a load model reproducing input/output length, concurrency, priority, and bursts..
- It reaches an acceptance decision through p50/p95 first-token and inter-token latency.
§ 03
Fix the boundary and acceptance criteria before implementation.
Distributed inference should make latency, throughput, and isolation predictable for real traffic—not merely maximize GPU utilization.
- 01
Stage 1
Build a load model reproducing input/output length, concurrency, priority, and bursts.
Review artifact 1 - 02
Stage 2
Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
Review artifact 2 - 03
Stage 3
Configure cache-aware routing, admission control, and preemption by SLO class.
Review artifact 3 - 04
Stage 4
Inject node, link, and worker failures to measure draining, retry, and recovery.
Review artifact 4
§ 04
Hypothetical workloads make the applicability boundary concrete.
Multiple models or tenants share a constrained GPU pool
Optimizing average token throughput can let long requests displace short-request time to first token.
- APPROACH
- Build a load model reproducing input/output length, concurrency, priority, and bursts.
- BOUNDARY
- Synthetic peak throughput is not presented as production SLO performance.
Time-to-first-token, inter-token latency, and throughput SLOs must coexist
When scheduling and model parallelism are designed separately, failure and retry costs become unpredictable.
- APPROACH
- Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
- BOUNDARY
- GPU utilization alone cannot represent user latency or reliability.
§ 05
Review gains and costs in the same table.
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| Multiple models or tenants share a constrained GPU pool | Build a load model reproducing input/output length, concurrency, priority, and bursts. | Synthetic peak throughput is not presented as production SLO performance. | p50/p95 first-token and inter-token latency |
| Time-to-first-token, inter-token latency, and throughput SLOs must coexist | Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape. | GPU utilization alone cannot represent user latency or reliability. | Goodput and queue time |
Distributed GPU Serving & Scheduling: system plate
- A single low-traffic model fits comfortably on one machine
Synthetic peak throughput is not presented as production SLO performance.
- SIGNAL
- p50/p95 first-token and inter-token latency
- MITIGATION
- Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
- Real request distributions and SLOs are undefined
GPU utilization alone cannot represent user latency or reliability.
- SIGNAL
- Goodput and queue time
- MITIGATION
- Configure cache-aware routing, admission control, and preemption by SLO class.
- The workflow begins with Build a load model reproducing input/output length, concurrency, priority, and bursts..
- It reaches an acceptance decision through p50/p95 first-token and inter-token latency.
§ 07
Agree on measurement conditions before publishing a result.
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| p50/p95 first-token and inter-token latency | Build a load model reproducing input/output length, concurrency, priority, and bursts. | Repeated runs satisfy the acceptance threshold agreed during discovery | Synthetic peak throughput is not presented as production SLO performance. |
| Goodput and queue time | Compare tensor, pipeline, data parallelism, and prefill/decode separation by model shape. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Isolation and recovery time | Configure cache-aware routing, admission control, and preemption by SLO class. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
§ 08
Conditions for not applying the capability are part of the design.
A single low-traffic model fits comfortably on one machine
Synthetic peak throughput is not presented as production SLO performance.
Real request distributions and SLOs are undefined
GPU utilization alone cannot represent user latency or reliability.
§ 09
Proceed through diagnosis, design, and validation gates.
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
p50/p95 first-token and inter-token latency - 02
Design
PattyCompare tensor, pipeline, data parallelism, and prefill/decode separation by model shape.
ClientConfirm owners and acceptance criteria.
Goodput and queue time - 03
Validation
PattyConfigure cache-aware routing, admission control, and preemption by SLO class.
ClientMake the production-transition or stop decision.
Isolation and recovery time
§ 10
Artifacts remain with the operating organization after the engagement.
- Distributed GPU Serving & Scheduling decision record
- Distributed inference should make latency, throughput, and isolation predictable for real traffic—not merely maximize GPU utilization.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- p50/p95 first-token and inter-token latency · Goodput and queue time · Isolation and recovery timeJointly maintained
- Operations and recovery runbook
- Synthetic peak throughput is not presented as production SLO performance. · GPU utilization alone cannot represent user latency or reliability.Operating-team owned
§ 11
Use shared terms with explicit operating meaning.
- Distributed GPU Serving & Scheduling
- Model request shapes, parallelism, batching, routing, and failure domains together to engineer service levels.
- Acceptance criterion
- p50/p95 first-token and inter-token latency
- Operating boundary
- Synthetic peak throughput is not presented as production SLO performance.
REFERENCES
References and primary material
- vLLM documentation
Primary material for the method and terminology.
- Ray Serve LLM
Primary material for the method and terminology.
Begin by determining whether Distributed GPU Serving & Scheduling is the justified next step.
We define scope and validation against representative work, data and infrastructure boundaries, and explicit failure conditions.