X-MoE-4D: Breaking the Scaling Barrier for Trillion Parameter MoEs on Hierarchical HPC Systems
Published in SC 2026, 2026
Accepted to SC26 (The International Conference for High Performance Computing, Networking, Storage, and Analysis).
Joint work between the University of Illinois Urbana-Champaign and Oak Ridge National Laboratory (ORNL).
X-MoE-4D scales Mixture-of-Experts training to trillion-parameter models on hierarchical HPC systems.
Recommended citation: Z. Wang, L. Jing, E. Herron, S. Dash, F. Wang, M. Zhang. "X-MoE-4D: Breaking the Scaling Barrier for Trillion Parameter MoEs on Hierarchical HPC Systems." The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC26), 2026.