ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

4 October 2019

Yuxiong He

Papers citing "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models"

50 / 164 papers shown

Title
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 229 0 0 08 Jan 2025
Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning Lang Xu Quentin G. Anthony Jacob Hatef Hari Subramoni Hari Subramoni Dhabaleswar K. Panda 37 0 0 08 Jan 2025
SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment Yuchun Fan Yongyu Mu Yilin Wang Lei Huang Junhao Ruan Yangqiu Song Tong Xiao Shujian Huang Xiaocheng Feng Jingbo Zhu LRM 49 4 0 08 Jan 2025
Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap Hyunwoo Ko Guijin Son Dasol Choi RALM LRM 80 9 0 05 Jan 2025
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs LLM-jp Akiko Aizawa Eiji Aramaki Bowen Chen Fei Cheng ... Yuya Yamamoto Yusuke Yamauchi Hitomi Yanaka Rio Yokota Koichiro Yoshino 57 14 0 31 Dec 2024
Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism Tim Tsz-Kit Lau Weijian Li Chenwei Xu Han Liu Mladen Kolar 200 0 0 30 Dec 2024
Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation Samin Mahdizadeh Sani Pouya Sadeghi Thuy-Trang Vu Yadollah Yaghoobzadeh Gholamreza Haffari 78 2 0 17 Dec 2024
EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models Jialiang Cheng Ning Gao Yun Yue Zhiling Ye Jiadi Jiang Jian Sha OffRL 82 0 0 10 Dec 2024
Libra: Leveraging Temporal Images for Biomedical Radiology Analysis Xi Zhang Zaiqiao Meng Jake Lever Edmond S. L. Ho MedIm 101 1 0 28 Nov 2024
Distributed Sign Momentum with Local Steps for Training Transformers Shuhua Yu Ding Zhou Cong Xie An Xu Zhi-Li Zhang Xin Liu S. Kar 69 0 0 26 Nov 2024
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection Jinqi Xiao S. Sang Tiancheng Zhi Jing Liu Qing Yan Linjie Luo Bo Yuan Bo Yuan VLM 103 1 0 26 Nov 2024
Large Language Model with Region-guided Referring and Grounding for CT Report Generation Zhaoyu Chen Yequan Bie Haibo Jin Hao Chen 235 0 0 23 Nov 2024
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training Jared Fernandez Luca Wehrstedt Leonid Shamis Mostafa Elhoushi Kalyan Saladi Yonatan Bisk Emma Strubell Jacob Kahn 254 3 0 20 Nov 2024
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues Riccardo Grazzi Julien N. Siems Jörg Franke Arber Zela Frank Hutter Massimiliano Pontil 92 11 0 19 Nov 2024
LaVin-DiT: Large Vision Diffusion Transformer Zhaoqing Wang Xiaobo Xia Runnan Chen Dongdong Yu Changhu Wang Mingming Gong Tongliang Liu 100 6 0 18 Nov 2024
Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey Zhihong Liu Xin Xu Peng Qiao Dongsheng Li OffRL 27 2 0 08 Nov 2024
Exploring Hierarchical Molecular Graph Representation in Multimodal LLMs Chengxin Hu Hao Li Yihe Yuan Jing Li Ivor Tsang 51 0 0 07 Nov 2024
MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs Sheng-Chieh Lin Chankyu Lee M. Shoeybi Jimmy J. Lin Bryan Catanzaro Ming-Yu Liu 73 12 0 04 Nov 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 84 5 0 28 Oct 2024
Cross-lingual Transfer of Reward Models in Multilingual Alignment Jiwoo Hong Noah Lee Rodrigo Martínez-Castaño César Rodríguez James Thorne 48 4 0 23 Oct 2024
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models Michael Noukhovitch Shengyi Huang Sophie Xhonneux Arian Hosseini Rishabh Agarwal Rameswar Panda OffRL 85 6 0 23 Oct 2024
Scaling Diffusion Language Models via Adaptation from Autoregressive Models Shansan Gong Shivam Agarwal Yizhe Zhang Jiacheng Ye Lin Zheng ... Peilin Zhao W. Bi Jiawei Han Hao Peng Lingpeng Kong AI4CE 78 16 0 23 Oct 2024
Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models Yuheng Lu Bingshuo Qian Caixia Yuan Huixing Jiang Xiaojie Wang CLL 34 0 0 22 Oct 2024
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 73 5 0 22 Oct 2024
ToW: Thoughts of Words Improve Reasoning in Large Language Models Zhikun Xu Ming shen Jacob Dineen Zhaonan Li Xiao Ye Shijie Lu Aswin Rrv Chitta Baral Ben Zhou LRM 200 1 0 21 Oct 2024
Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens Zhepeng Cen Yao Liu Siliang Zeng Pratik Chaudhar Huzefa Rangwala George Karypis Rasool Fakoor SyDa AIFin 34 3 0 18 Oct 2024
Scalable Multi-Domain Adaptation of Language Models using Modular Experts Peter Schafhalter Shun Liao Yanqi Zhou Chih-Kuan Yeh Arun Kandoor James Laudon MoE 34 1 0 14 Oct 2024
Data Selection via Optimal Control for Language Models Yuxian Gu Li Dong Hongning Wang Y. Hao Qingxiu Dong Furu Wei Minlie Huang AI4CE 58 5 0 09 Oct 2024
Parameter Efficient Fine-tuning via Explained Variance Adaptation Fabian Paischer Lukas Hauzenberger Thomas Schmied Benedikt Alkin Marc Peter Deisenroth Sepp Hochreiter 42 4 0 09 Oct 2024
CursorCore: Assist Programming through Aligning Anything Hao Jiang Qi Liu Rui Li Shengyu Ye Shijin Wang 53 1 0 09 Oct 2024
Aria: An Open Multimodal Native Mixture-of-Experts Model Dongxu Li Yudong Liu Haoning Wu Yue Wang Zhiqi Shen ... Lihuan Zhang Hanshu Yan Guoyin Wang Bei Chen Junnan Li MoE 59 48 0 08 Oct 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 197 0 0 07 Oct 2024
Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation Jing Yuan Luo Yunlong Song Victor Klemm Fan Shi Davide Scaramuzza Marco Hutter 33 2 0 04 Oct 2024
ToolGen: Unified Tool Retrieval and Calling via Generation Renxi Wang Xudong Han Lei Ji Shu Wang Timothy Baldwin Haonan Li LLMAG 75 6 0 04 Oct 2024
Mixture of Diverse Size Experts Manxi Sun Wei Liu Jian Luan Pengzhi Gao Bin Wang MoE 28 1 0 18 Sep 2024
MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model Junjie Li Yang Liu Weiqing Liu Shikai Fang Lewen Wang Chang Xu Jiang Bian VGen 46 4 0 04 Sep 2024
Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer Jinghan Yao Sam Ade Jacobs Masahiro Tanaka Olatunji Ruwase Hari Subramoni D. Panda 33 2 0 30 Aug 2024
IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities Bin Wang Chunyu Xie Dawei Leng Yuhui Yin MLLM 54 1 0 23 Aug 2024
GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models Kunsheng Tang Wenbo Zhou Jie Zhang Aishan Liu Gelei Deng Shuai Li Peigui Qi Weiming Zhang Tianwei Zhang Nenghai Yu 46 3 0 22 Aug 2024
Tamper-Resistant Safeguards for Open-Weight LLMs Rishub Tamirisa Bhrugu Bharathi Long Phan Andy Zhou Alice Gatti ... Andy Zou Dawn Song Bo Li Dan Hendrycks Mantas Mazeika AAML MU 53 43 0 01 Aug 2024
Meltemi: The first open Large Language Model for Greek Leon Voukoutis Dimitris Roussis Georgios Paraskevopoulos Sokratis Sofianopoulos Prokopis Prokopidis Vassilis Papavasileiou Athanasios Katsamanis Stelios Piperidis Vassilis Katsouros VLM 41 8 0 30 Jul 2024
OmniBal: Towards Fast Instruct-tuning for Vision-Language Models via Omniverse Computation Balance Yongqiang Yao Jingru Tan Jiahao Hu Feizhao Zhang Xin Jin ... Ruihao Gong Pengfei Liu Pengfei Liu Dahua Lin Ningyi Xu VLM 52 1 0 30 Jul 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 61 9 0 24 Jul 2024
Weighted Grouped Query Attention in Transformers Sai Sena Chinnakonduru Astarag Mohapatra 44 5 0 15 Jul 2024
GOFA: A Generative One-For-All Model for Joint Graph Language Modeling Lecheng Kong Jiarui Feng Hao Liu Chengsong Huang Jiaxin Huang Yixin Chen Muhan Zhang AI4CE 77 8 0 12 Jul 2024
Training on the Test Task Confounds Evaluation and Emergence Ricardo Dominguez-Olmedo Florian E. Dorner Moritz Hardt ELM 71 7 1 10 Jul 2024
Controlling Space and Time with Diffusion Models Daniel Watson Saurabh Saxena Lala Li Andrea Tagliasacchi David J. Fleet VGen 71 27 0 10 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 42 43 0 09 Jul 2024
MST5 -- Multilingual Question Answering over Knowledge Graphs Nikit Srivastava Mengshi Ma Daniel Vollmers Hamada M. Zahera Diego Moussallem A. N. Ngomo 34 0 0 08 Jul 2024
Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation Yuchen Yang Yingdong Shi Cheems Wang Xiantong Zhen Yuxuan Shi Jun Xu 40 1 0 24 Jun 2024