Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models
Xing Chen and Hengshuai Yao separate two effects that are conflated when you reduce top-k at MoE inference time, and recover most of the lost accuracy with a single integer and no training.
Ask this paper
The conflation: Router probability renormalization implicitly calibrates expert output gain to the training top-k. Reducing k at inference changes both which experts run and how strongly the expert branch contributes.
The fix: Activate the top k1 experts but normalize by the probability mass of the top k2 experts. That is one integer, no parameters, no training, and no measurable compute overhead.
Numbers: On Qwen3.6-35B-A3B, going from 8 to 4 experts costs 4.65 MMLU points under standard renormalization and 0.35 points with k2=16, while halving routed-expert compute. On the 11x larger Qwen3.5-397B-A17B, 10 to 5 experts costs 0.55 points with an appropriate reference set.
Two cautions from the analysis: Removing renormalization entirely is catastrophic, so preserving a suitable reference mass is required. Perplexity and downstream accuracy prefer different k2, so selecting MoE compression settings from unlabeled text alone will pick the wrong value.
Abstract
Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the training top-$k$: reducing $k$ at inference changes not only which experts are used but also the strength of the expert branch. We separate these effects by activating the top $k_1$ experts while normalizing by the probability mass of the top $k_2$ experts, introducing one integer with no parameters, training, or measurable compute overhead. On Qwen3.6-35B-A3B, reducing from 8 to 4 experts causes a 4.65-point MMLU drop under standard renormalization but only 0.35 points with $k_2=16$, while halving routed-expert compute. The result replicates on the $11\times$ larger Qwen3.5-397B-A17B, where reducing from 10 to 5 experts loses only 0.55 points with an appropriate reference set. Removing renormalization entirely is catastrophic, showing that preserving a suitable reference mass is crucial. We further find that perplexity and downstream accuracy favor different $k_2$, cautioning against selecting MoE compression settings using unlabeled text alone. Analyses also show that expert identity matters substantially more than expert weighting, while balanced and domain-specialized routing leaves limited room for expert pruning.