HOPE Prunes Mixture-of-Experts Models by Preserving Expert Cooperation

Researchers have introduced HOPE, a higher-order expert pruning method for mixture-of-experts language models, addressing a major memory and deployment bottleneck in large MoE systems. While MoE architectures allow models to scale efficiently by activating only a small subset of experts per token, most expert parameters remain resident in memory even when inactive. Expert pruning reduces this overhead by permanently removing low-utility experts, but conventional methods score each expert independently and can damage performance when cooperative expert groups are disrupted. The new method, named Higher-Order Pruning of Experts, evaluates pairwise expert interactions rather than treating experts in isolation. Researchers observed that MoE experts frequently co-activate in stable clusters. Independent pruning can break these relationships, especially at aggressive compression levels. HOPE models these dependencies using a pairwise interaction matrix and formulates expert removal as a quadratic optimization problem that minimizes estimated pruning error over a calibration dataset. The diagonal of the matrix resembles scores used by REAP, a leading first-order pruning baseline, while off-diagonal terms capture cooperative expert relationships. In experiments across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE achieved the best average ranking among five pruning methods. Its advantage increased at higher pruning rates, where preserving expert cooperation becomes more important. At 50 percent pruning, HOPE reached an average rank of 1.58, compared with 2.42 for REAP. On SWE-Bench Pro, an agentic coding benchmark, a HOPE-pruned Qwen3.5-35B model scored 42.3, matching the unpruned model’s 41.8 score, while REAP dropped to 36.2. The work was presented by researchers including Alex M. Tseng, Prannay Kaul, Luca Zancato, Wei Xia and Stefano Soatto, with code released through an AWS Laboratories repository. The pipeline requires a single calibration pass, solves a per-layer optimization problem, and produces a standard smaller model checkpoint compatible with common inference stacks. HOPE’s strongest gains appear in 40 to 50 percent pruning scenarios. Its current implementation prunes each MoE layer independently and models only pairwise interactions, leaving room for future work on cross-layer and higher-order expert relationships.