MoE Serving (Wide EP)
MoE serving is primarily a design problem in token routing, expert imbalance, and collective-communication tails—not parameter count.
Co-design batching, routing, all-to-all communication, and failure scope for wide expert parallel deployments.
§ 01
The operating conditions that justify MoE Serving (Wide EP)
Even with balanced averages, token concentration on a few experts makes every rank wait for the slowest.
Adding GPUs distributes memory while expanding communication and failure domains.
- The total model is large relative to active parameters and single-node placement is inefficient
- Expert-parallel width can be validated against real token distributions
- Network topology and collectives cannot be controlled
- At low traffic, complexity costs exceed memory benefits
MoE Serving (Wide EP): system plate
System 1
Profile expert selection and token imbalance by prompt class.
System 2
Compare intra-node and inter-node EP/TP placement against physical topology.
System 3
Measure all-to-all volume, overlappable compute, and straggler impact by stage.
System 4
Validate request scope and recovery strategy under expert, rank, and link failures.
- N1 N2context
- N2 N3decision
- N3 N4evidence
- The workflow begins with Profile expert selection and token imbalance by prompt class..
- It reaches an acceptance decision through Expert load balance.
§ 03
Fix the boundary and acceptance criteria before implementation.
MoE serving is primarily a design problem in token routing, expert imbalance, and collective-communication tails—not parameter count.
- 01
Stage 1
Profile expert selection and token imbalance by prompt class.
Review artifact 1 - 02
Stage 2
Compare intra-node and inter-node EP/TP placement against physical topology.
Review artifact 2 - 03
Stage 3
Measure all-to-all volume, overlappable compute, and straggler impact by stage.
Review artifact 3 - 04
Stage 4
Validate request scope and recovery strategy under expert, rank, and link failures.
Review artifact 4
§ 04
Hypothetical workloads make the applicability boundary concrete.
The total model is large relative to active parameters and single-node placement is inefficient
Even with balanced averages, token concentration on a few experts makes every rank wait for the slowest.
- APPROACH
- Profile expert selection and token imbalance by prompt class.
- BOUNDARY
- Throughput figures are not transferred across different hardware, models, or traffic.
Expert-parallel width can be validated against real token distributions
Adding GPUs distributes memory while expanding communication and failure domains.
- APPROACH
- Compare intra-node and inter-node EP/TP placement against physical topology.
- BOUNDARY
- Wide EP is not universally superior to TP or narrower EP.
§ 05
Review gains and costs in the same table.
| Decision | Gain | Cost | Watch |
|---|---|---|---|
| The total model is large relative to active parameters and single-node placement is inefficient | Profile expert selection and token imbalance by prompt class. | Throughput figures are not transferred across different hardware, models, or traffic. | Expert load balance |
| Expert-parallel width can be validated against real token distributions | Compare intra-node and inter-node EP/TP placement against physical topology. | Wide EP is not universally superior to TP or narrower EP. | Collective share and tail time |
MoE Serving (Wide EP): system plate
| Item | Method | Evidence | Boundary |
|---|---|---|---|
| Layer 1 | Profile expert selection and token imbalance by prompt class. | Expert load balance | Throughput figures are not transferred across different hardware, models, or traffic. |
| Layer 2 | Compare intra-node and inter-node EP/TP placement against physical topology. | Collective share and tail time | Wide EP is not universally superior to TP or narrower EP. |
| Layer 3 | Measure all-to-all volume, overlappable compute, and straggler impact by stage. | Token goodput and recovery scope | Throughput figures are not transferred across different hardware, models, or traffic. |
- The workflow begins with Profile expert selection and token imbalance by prompt class..
- It reaches an acceptance decision through Expert load balance.
§ 07
Agree on measurement conditions before publishing a result.
| Measure | Method | Pass condition | Caveat |
|---|---|---|---|
| Expert load balance | Profile expert selection and token imbalance by prompt class. | Repeated runs satisfy the acceptance threshold agreed during discovery | Throughput figures are not transferred across different hardware, models, or traffic. |
| Collective share and tail time | Compare intra-node and inter-node EP/TP placement against physical topology. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
| Token goodput and recovery scope | Measure all-to-all volume, overlappable compute, and straggler impact by stage. | Repeated runs satisfy the acceptance threshold agreed during discovery | — |
§ 08
Conditions for not applying the capability are part of the design.
Network topology and collectives cannot be controlled
Throughput figures are not transferred across different hardware, models, or traffic.
At low traffic, complexity costs exceed memory benefits
Wide EP is not universally superior to TP or narrower EP.
§ 09
Proceed through diagnosis, design, and validation gates.
- 01
Diagnosis
PattyAnalyze the current system and its failure signals.
ClientProvide representative work, data boundaries, and operating constraints.
Expert load balance - 02
Design
PattyCompare intra-node and inter-node EP/TP placement against physical topology.
ClientConfirm owners and acceptance criteria.
Collective share and tail time - 03
Validation
PattyMeasure all-to-all volume, overlappable compute, and straggler impact by stage.
ClientMake the production-transition or stop decision.
Token goodput and recovery scope
§ 10
Artifacts remain with the operating organization after the engagement.
- MoE Serving (Wide EP) decision record
- MoE serving is primarily a design problem in token routing, expert imbalance, and collective-communication tails—not parameter count.Client-owned · Patty-reviewed
- Validation harness and acceptance criteria
- Expert load balance · Collective share and tail time · Token goodput and recovery scopeJointly maintained
- Operations and recovery runbook
- Throughput figures are not transferred across different hardware, models, or traffic. · Wide EP is not universally superior to TP or narrower EP.Operating-team owned
§ 11
Use shared terms with explicit operating meaning.
- MoE Serving (Wide EP)
- Co-design batching, routing, all-to-all communication, and failure scope for wide expert parallel deployments.
- Acceptance criterion
- Expert load balance
- Operating boundary
- Throughput figures are not transferred across different hardware, models, or traffic.
REFERENCES
References and primary material
- DeepEP
Primary material for the method and terminology.
- vLLM Expert Parallel Deployment
Primary material for the method and terminology.
Begin by determining whether MoE Serving (Wide EP) is the justified next step.
We define scope and validation against representative work, data and infrastructure boundaries, and explicit failure conditions.