MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture
of Shards

MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards

1 October 2024

Liheng Chen

Lingpeng Kong

Chuan Wu

ArXiv (abs)PDF HTML

Papers citing "MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards"

17 / 17 papers shown

Title
AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robots Zhaxizhuoma Pengan Chen Ziniu Wu Jiawei Sun Dong Wang Peng Zhou Nieqing Cao Yan Ding Bin Zhao Xuelong Li 106 5 0 18 Sep 2024
MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts Lin Ning Harsh Lara Meiqi Guo Abhinav Rastogi MoMe MoE 66 2 0 02 Aug 2024
LoRA Meets Dropout under a Unified Framework Sheng Wang Liheng Chen Jiyue Jiang Boyang Xue Lingpeng Kong Chuan Wu 68 15 0 25 Feb 2024
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models Zhihong Shao Peiyi Wang Qihao Zhu Runxin Xu Jun-Mei Song ... Haowei Zhang Mingchuan Zhang Yiming Li Yu-Huan Wu Daya Guo ReLM LRM 167 1,288 0 05 Feb 2024
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,761 0 15 Mar 2023
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 231 3,158 0 20 Oct 2022
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them Mirac Suzgun Nathan Scales Nathanael Scharli Sebastian Gehrmann Yi Tay ... Aakanksha Chowdhery Quoc V. Le Ed H. Chi Denny Zhou Jason W. Wei ALM ELM LRM ReLM 271 1,139 0 17 Oct 2022
DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation Mojtaba Valipour Mehdi Rezagholizadeh I. Kobyzev A. Ghodsi 101 183 0 14 Oct 2022
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models Aarohi Srivastava Abhinav Rastogi Abhishek Rao Abu Awal Md Shoeb Abubakar Abid ... Zhuoye Zhao Zijian Wang Zijie J. Wang Zirui Wang Ziyi Wu ELM 213 1,777 0 09 Jun 2022
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks Yizhong Wang Swaroop Mishra Pegah Alipoormolabashi Yeganeh Kordi Amirreza Mirzaei ... Chitta Baral Yejin Choi Noah A. Smith Hannaneh Hajishirzi Daniel Khashabi ELM 123 859 0 16 Apr 2022
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation Tao Ge Si-Qing Chen Furu Wei MoE 79 23 0 16 Feb 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 853 9,714 0 28 Jan 2022
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 356 4,598 0 27 Oct 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 238 5,665 0 07 Jul 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 502 10,526 0 17 Jun 2021
Lessons on Parameter Sharing across Layers in Transformers Sho Takase Shun Kiyono 58 87 0 13 Apr 2021
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages J. Clark Eunsol Choi Michael Collins Dan Garrette Tom Kwiatkowski Vitaly Nikolaev J. Palomaki 189 613 0 10 Mar 2020