Title
MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production C. Jin Ziheng Jiang Zhihao Bai Zheng Zhong J. Liu ... Yanghua Peng Xuanzhe Liu Xuanzhe Liu Xin Jin Xin Liu MoE 7 0 0 16 May 2025
Aquarius: A Family of Industry-Level Video Generation Models for Marketing Scenarios Huafeng Shi Jianzhong Liang Rongchang Xie Xian Wu Cheng Chen Chang Liu VGen 17 0 0 14 May 2025
Understanding Stragglers in Large Model Training Using What-if Analysis Jinkun Lin Ziheng Jiang Zuquan Song Sida Zhao Menghan Yu ... Shuguang Wang Yanghua Peng Xin Liu Aurojit Panda Jinyang Li 25 0 0 09 May 2025
Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs Yehui Tang Yichun Yin Yaoyuan Wang Hang Zhou Yu Pan ... Zhe Liu Zhicheng Liu Z. Tu Zilin Ding Zongyuan Zhan MoE 32 0 0 07 May 2025
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code Kazuki Fujii Yukito Tajima Sakae Mizuki Hinari Shimada Taihei Shiotani ... Kakeru Hattori Youmi Ma Hiroya Takamura Rio Yokota Naoaki Okazaki SyDa 49 0 0 05 May 2025
Nesterov Method for Asynchronous Pipeline Parallel Optimization Thalaiyasingam Ajanthan Sameera Ramasinghe Yan Zuo Gil Avraham Alexander Long 24 0 0 02 May 2025
Galvatron: An Automatic Distributed System for Efficient Foundation Model Training Xinyi Liu Y. Wang Shenhan Zhu Fangcheng Fu Qingshuo Liu Guangming Lin Bin Cui GNN 140 0 0 30 Apr 2025
Towards Easy and Realistic Network Infrastructure Testing for Large-scale Machine Learning Jinsun Yoo ChonLam Lao Lianjie Cao Bob Lantz Minlan Yu Tushar Krishna Puneet Sharma 52 0 0 29 Apr 2025
Taming the Titans: A Survey of Efficient LLM Inference Serving Ranran Zhen J. Li Yixin Ji Z. Yang Tong Liu Qingrong Xia Xinyu Duan Z. Wang Baoxing Huai M. Zhang LLMAG 77 0 0 28 Apr 2025
Accelerating Mixture-of-Experts Training with Adaptive Expert Replication Athinagoras Skiadopoulos Mark Zhao Swapnil Gandhi Thomas Norrie Shrijeet Mukherjee Christos Kozyrakis MoE 91 0 0 28 Apr 2025
Energy Considerations of Large Language Model Inference and Efficiency Optimizations Jared Fernandez Clara Na Vashisth Tiwari Yonatan Bisk Sasha Luccioni Emma Strubell 41 0 0 24 Apr 2025
Trends in AI Supercomputers Konstantin Pilz James Sanders Robi Rahman Lennart Heim GNN ELM 29 0 0 22 Apr 2025
MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core Dennis Liu Zijie Yan Xin Yao Tong Liu V. Korthikanti ... Jiajie Yao Chandler Zhou David Wu Xipeng Li J. Yang MoE 65 0 0 21 Apr 2025
SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training Zheng Li Yong-Jin Liu Wei Zhang Tailing Yuan Bin Chen Chengru Song Di Zhang 34 0 0 20 Apr 2025
From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs Jiliang Ni Jiachen Pu Zhongyi Yang Kun Zhou Hui Wang Xiaoliang Xiao Dakui Wang Xin Li Jingfeng Luo Conggang Hu 34 0 0 18 Apr 2025
OVERLORD: Ultimate Scaling of DataLoader for Multi-Source Large Foundation Model Training Juntao Zhao Qi Lu Wei Jia Borui Wan Lei Zuo ... Size Zheng H. Lin Xin Liu Xin Liu Chuan Wu AI4CE 34 0 0 14 Apr 2025
Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training Mingyu Liang Hiwot Tadese Kassa Wenyin Fu Brian Coutinho Louis Feng Christina Delimitrou 28 0 0 12 Apr 2025
Jupiter: Fast and Resource-Efficient Collaborative Inference of Generative LLMs on Edge Devices Shengyuan Ye Bei Ouyang Liekang Zeng Tianyi Qian Xiaowen Chu Jian Tang Xu Chen 29 1 0 11 Apr 2025
Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents Yueying Li Jim Dai Tianyi Peng 126 1 0 10 Apr 2025
Kimi-VL Technical Report Kimi Team Angang Du B. Yin Bowei Xing Bowen Qu ... Zhiqi Huang Zihao Huang Zijia Zhao Zhengzhang Chen Zongyu Lin MLLM VLM MoE 204 2 0 10 Apr 2025
Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs Dongyang Fan Vinko Sabolčec Matin Ansaripour Ayush Kumar Tarun Martin Jaggi Antoine Bosselut Imanol Schlag 36 0 0 08 Apr 2025
TAGC: Optimizing Gradient Communication in Distributed Transformer Training Igor Polyakov Alexey Dukhanov Egor Spirin 41 0 0 08 Apr 2025
Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Training Daiyaan Arfeen Dheevatsa Mudigere Ankit More Bhargava Gopireddy Ahmet Inci G. R. Ganger 23 0 0 08 Apr 2025
PipeDec: Low-Latency Pipeline-based Inference with Dynamic Speculative Decoding towards Large-scale Models Haofei Yin Mengbai Xiao Rouzhou Lu Xiao Zhang Dongxiao Yu Guanghui Zhang AI4CE 24 0 0 05 Apr 2025
HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs Yongji Wu Xueshen Liu Shuowei Jin Ceyu Xu Feng Qian Ziming Mao Matthew Lentz Danyang Zhuo Ion Stoica MoMe MoE 61 0 0 04 Apr 2025
MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators Beichen Huang Yueming Yuan Zelei Shao Minjia Zhang MQ MoE 42 0 0 03 Apr 2025
LandMarkSystem Technical Report Zhenxiang Ma Zhenyu Yang Miao Tao Yuanzhen Zhou Zeyu He Yuchang Zhang Rong Fu Hengjie Li 3DGS 33 0 0 27 Mar 2025
Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization Zhanda Zhu Christina Giannoula Muralidhar Andoorveedu Qidong Su Karttikeya Mangalam Bojian Zheng Gennady Pekhimenko VLM MoE 51 0 0 24 Mar 2025
WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training Zhilin Wang Anna Cai Xinfeng Xie Zaifeng Pan Yue Guan ... Shikai Li Jianyu Huang Chris Cai Yuchen Hao Yufei Ding 39 2 0 23 Mar 2025
Training Video Foundation Models with NVIDIA NeMo Zeeshan Patel Ethan He Parth Mannan Xiaowei Ren Ryan Wolf ... Rong Ou Pallab Bhattacharya David Page Nima Tajbakhsh Ashwath Aithal VGen 43 0 0 17 Mar 2025
AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applications Haiying Shen Tanmoy Sen 44 0 0 17 Mar 2025
Mitigating KV Cache Competition to Enhance User Experience in LLM Inference Haiying Shen Tanmoy Sen Masahiro Tanaka 154 0 0 17 Mar 2025
The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation Olivier Gouvert Julie Hunter Jérôme Louradour Christophe Cerisara Evan Dufraisse Yaya Sy Laura Rivière Jean-Pierre Lorré OpenLLM-France community 159 0 0 15 Mar 2025
Automatic Operator-level Parallelism Planning for Distributed Deep Learning -- A Mixed-Integer Programming Approach Ruifeng She Bowen Pang Kai Li Zehua Liu Tao Zhong 61 0 0 12 Mar 2025
Position-Aware Depth Decay Decoding ( $D^3$ ): Boosting Large Language Model Inference Efficiency Siqi Fan Xuezhi Fang Xingrun Xing Peng Han Shuo Shang Yequan Wang 58 0 0 11 Mar 2025
Green Prompting Marta Adamska Daria Smirnova Hamid Nasiri Zhengxin Yu Peter Garraghan 160 0 0 09 Mar 2025
PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization Xinyi Wan Penghui Qi Guangxing Huang Jialin Li Min Lin 39 0 0 03 Mar 2025
ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs Hao Ge Junda Feng Qi Huang Fangcheng Fu Xiaonan Nie Lei Zuo Yanghua Peng Bin Cui Xin Liu 42 2 0 28 Feb 2025
Stealing Training Data from Large Language Models in Decentralized Training through Activation Inversion Attack Chenxi Dai Lin Lu Pan Zhou 50 0 0 22 Feb 2025
Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs Longxu Dou Qian Liu Fan Zhou Changyu Chen Zili Wang ... Tianyu Pang Chao Du Xinyi Wan Wei Lu Min Lin 106 1 0 18 Feb 2025
Understanding Silent Data Corruption in LLM Training Jeffrey Ma Hengzhi Pei Leonard Lausen George Karypis 37 0 0 17 Feb 2025
Energy-Conscious LLM Decoding: Impact of Text Generation Strategies on GPU Energy Consumption Alireza Nik Michael A. Riegler P. Halvorsen 46 0 0 17 Feb 2025
APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs Yuxiang Huang Mingye Li Xu Han Chaojun Xiao Weilin Zhao Sun Ao Hao Zhou Jie Zhou Zhiyuan Liu Maosong Sun 44 0 0 17 Feb 2025
Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization Bowen Pang Kai Li Ruifeng She Feifan Wang OffRL 43 2 0 14 Feb 2025
Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model Guoqing Ma Haoyang Huang K. Yan L. Chen Nan Duan ... Yixuan Wang Yuanwei Lu Yu-Cheng Chen Yu-Juan Luo Y. Luo DiffM VGen 175 17 0 14 Feb 2025
Importance Sampling via Score-based Generative Models Heasung Kim Taekyun Lee Hyeji Kim Gustavo de Veciana MedIm DiffM 136 0 0 07 Feb 2025
M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference Nikhil Bhendawade Mahyar Najibi Devang Naik Irina Belousova MoE 85 0 0 04 Feb 2025
GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments Yanyu Chen Ganhong Huang 108 0 0 28 Jan 2025
A Survey on Memory-Efficient Large-Scale Model Training in AI for Science Kaiyuan Tian Linbo Qiao Baihui Liu Gongqingjian Jiang Dongsheng Li 36 0 0 21 Jan 2025
Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning Lang Xu Quentin G. Anthony Jacob Hatef Hari Subramoni Hari Subramoni Dhabaleswar K. Panda 37 0 0 08 Jan 2025