$μ$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these rely on calibration data, domain shift may arise for unknown downstream tasks. With a computationally efficient calibration, activation-aware pruning can be executed for every prompt adaptively, yet achieving reduced complexity at inference. We formulate it as a mixture of micro-experts, called $\mu$-MoE. Several experiments demonstrate that $\mu$-MoE can dynamically adapt to task/prompt-dependent structured sparsity on the fly.

@article{koike-akino2025_2505.18451,
  title={ $μ$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts },
  author={ Toshiaki Koike-Akino and Jing Liu and Ye Wang },
  journal={arXiv preprint arXiv:2505.18451},
  year={ 2025 }
}