MoE Serving (Wide EP)
MoE serving is primarily a design problem in token routing, expert imbalance, and collective-communication tails—not parameter count.
Co-design batching, routing, all-to-all communication, and failure scope for wide expert parallel deployments.
01
Problem definition
Even with balanced averages, token concentration on a few experts makes every rank wait for the slowest.
Adding GPUs distributes memory while expanding communication and failure domains.
- The total model is large relative to active parameters and single-node placement is inefficient
- Expert-parallel width can be validated against real token distributions
- Network topology and collectives cannot be controlled
- At low traffic, complexity costs exceed memory benefits
MoE Serving (Wide EP): system plate
System 1
Profile expert selection and token imbalance by prompt class.
System 2
Compare intra-node and inter-node EP/TP placement against physical topology.
System 3
Measure all-to-all volume, overlappable compute, and straggler impact by stage.
System 4
Validate request scope and recovery strategy under expert, rank, and link failures.
- N1 N2context
- N2 N3decision
- N3 N4evidence
- The workflow begins with Profile expert selection and token imbalance by prompt class..
- It reaches an acceptance decision through Expert load balance.
03
Design method
MoE serving is primarily a design problem in token routing, expert imbalance, and collective-communication tails—not parameter count.
- 01
Stage 1
Profile expert selection and token imbalance by prompt class.
- 02
Stage 2
Compare intra-node and inter-node EP/TP placement against physical topology.
- 03
Stage 3
Measure all-to-all volume, overlappable compute, and straggler impact by stage.
- 04
Stage 4
Validate request scope and recovery strategy under expert, rank, and link failures.
04
Application scenarios
The total model is large relative to active parameters and single-node placement is inefficient
Even with balanced averages, token concentration on a few experts makes every rank wait for the slowest.
- APPROACH
- Profile expert selection and token imbalance by prompt class.
- BOUNDARY
- Throughput figures are not transferred across different hardware, models, or traffic.
Expert-parallel width can be validated against real token distributions
Adding GPUs distributes memory while expanding communication and failure domains.
- APPROACH
- Compare intra-node and inter-node EP/TP placement against physical topology.
- BOUNDARY
- Wide EP is not universally superior to TP or narrower EP.
05
Design choices
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| The total model is large relative to active parameters and single-node placement is inefficient | Profile expert selection and token imbalance by prompt class. | Throughput figures are not transferred across different hardware, models, or traffic. | Expert load balance |
| Expert-parallel width can be validated against real token distributions | Compare intra-node and inter-node EP/TP placement against physical topology. | Wide EP is not universally superior to TP or narrower EP. | Collective share and tail time |
MoE Serving (Wide EP): system plate
| Item | Method | Evidence | Boundary |
|---|---|---|---|
| Layer 1 | Profile expert selection and token imbalance by prompt class. | Expert load balance | Throughput figures are not transferred across different hardware, models, or traffic. |
| Layer 2 | Compare intra-node and inter-node EP/TP placement against physical topology. | Collective share and tail time | Wide EP is not universally superior to TP or narrower EP. |
| Layer 3 | Measure all-to-all volume, overlappable compute, and straggler impact by stage. | Token goodput and recovery scope | Throughput figures are not transferred across different hardware, models, or traffic. |
- The workflow begins with Profile expert selection and token imbalance by prompt class..
- It reaches an acceptance decision through Expert load balance.
07
Validation plan
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| Expert load balance | Profile expert selection and token imbalance by prompt class. | Repeated runs satisfy the acceptance threshold agreed during discovery | Throughput figures are not transferred across different hardware, models, or traffic. |
| Collective share and tail time | Compare intra-node and inter-node EP/TP placement against physical topology. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Token goodput and recovery scope | Measure all-to-all volume, overlappable compute, and straggler impact by stage. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
08
Constraints and failure conditions
Network topology and collectives cannot be controlled
Throughput figures are not transferred across different hardware, models, or traffic.
At low traffic, complexity costs exceed memory benefits
Wide EP is not universally superior to TP or narrower EP.
09
Engagement model
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
Expert load balance - 02
Design
PattyCompare intra-node and inter-node EP/TP placement against physical topology.
ClientConfirm owners and acceptance criteria.
Collective share and tail time - 03
Validation
PattyMeasure all-to-all volume, overlappable compute, and straggler impact by stage.
ClientMake the production-transition or stop decision.
Token goodput and recovery scope
10
Durable deliverables
- MoE Serving (Wide EP) decision record
- MoE serving is primarily a design problem in token routing, expert imbalance, and collective-communication tails—not parameter count.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- Expert load balance · Collective share and tail time · Token goodput and recovery scopeJointly maintained
- Operations and recovery runbook
- Throughput figures are not transferred across different hardware, models, or traffic. · Wide EP is not universally superior to TP or narrower EP.Operating-team owned
11
Terminology
- MoE Serving (Wide EP)
- Co-design batching, routing, all-to-all communication, and failure scope for wide expert parallel deployments.
- Acceptance criterion
- Expert load balance
- Operating boundary
- Throughput figures are not transferred across different hardware, models, or traffic.
REFERENCES
References and primary material
- DeepEP
Primary material for the method and terminology.
- vLLM Expert Parallel Deployment
Primary material for the method and terminology.