When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

27 February 2024

Papers citing "When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method"

26 / 76 papers shown

Title
Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning Yun-Da Tsai Mingjie Liu Haoxing Ren SyDa 31 9 0 06 Jul 2024
AI Safety in Generative AI Large Language Models: A Survey Jaymari Chua Yun Yvonna Li Shiyi Yang Chen Wang Lina Yao LM&MA 36 12 0 06 Jul 2024
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale Wenzhen Zheng Wenbo Pan Xu Xu Libo Qin Li Yue Ming Zhou CLL 34 6 0 02 Jul 2024
Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts Junmo Kang Leonid Karlinsky Hongyin Luo Zhen Wang Jacob A. Hansen James Glass David D. Cox Rameswar Panda Rogerio Feris Alan Ritter MoMe MoE 36 8 0 17 Jun 2024
Large Scale Transfer Learning for Tabular Data via Language Modeling Josh Gardner Juan C. Perdomo Ludwig Schmidt LMTD 36 13 0 17 Jun 2024
Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent Lin Wang Zhichao Wang Xiaoying Tang 45 1 0 17 Jun 2024
Beyond Model Collapse: Scaling Up with Synthesized Data Requires Reinforcement Yunzhen Feng Elvis Dohmatob Pu Yang Francois Charton Julia Kempe 53 17 0 11 Jun 2024
Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe Alicja Ziarko Albert Q. Jiang Bartosz Piotrowski Wenda Li M. Jamnik Piotr Miłoś 34 0 0 06 Jun 2024
D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models Haoran Que Jiaheng Liu Ge Zhang Chenchen Zhang Xingwei Qu ... Jie Fu Wenbo Su Jiamang Wang Lin Qu Bo Zheng CLL 38 13 0 03 Jun 2024
A Survey of Multimodal Large Language Model from A Data-centric Perspective Tianyi Bai Hao Liang Binwang Wan Yanran Xu Xi Li ... Ping-Chia Huang Jiulong Shan Conghui He Binhang Yuan Wentao Zhang 52 36 0 26 May 2024
LoRA Learns Less and Forgets Less D. Biderman Jose Javier Gonzalez Ortiz Jacob P. Portes Mansheej Paul Philip Greengard ... Sam Havens Vitaliy Chiley Jonathan Frankle Cody Blakeney John P. Cunningham CLL 35 110 0 15 May 2024
High-level Stream Processing: A Complementary Analysis of Fault Recovery Adriano Vogel Sören Henning Esteban Perez-Wohlfeil Otmar Ertl Rick Rabiser 40 1 0 13 May 2024
SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts R. Prabhakar R. Sivaramakrishnan Darshan Gandhi Yun Du Mingran Wang ... Urmish Thakker Dawei Huang Sumti Jairath Kevin J. Brown K. Olukotun MoE 39 12 0 13 May 2024
Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice? D. Zhu Pinzhen Chen Miaoran Zhang Barry Haddow Xiaoyu Shen Dietrich Klakow 46 9 0 22 Apr 2024
BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models Qi Luo Hengxu Yu Xiao Li 44 1 0 03 Apr 2024
CodeS: Natural Language to Code Repository via Multi-Layer Sketch Daoguang Zan Ailun Yu Wei Liu Dong Chen Bo Shen ... Bei Guan Zhiguang Yang Yongji Wang Qianxiang Wang Li-zhen Cui 33 14 0 25 Mar 2024
Dial-insight: Fine-tuning Large Language Models with High-Quality Domain-Specific Data Preventing Capability Collapse Jianwei Sun Chaoyang Mei Linlin Wei Kaiyu Zheng Na Liu Ming Cui Tianyi Li ALM 40 4 0 14 Mar 2024
Selecting Large Language Model to Fine-tune via Rectified Scaling Law Haowei Lin Baizhou Huang Haotian Ye Qinyu Chen Zihao Wang Sujian Li Jianzhu Ma Xiaojun Wan James Zou Yitao Liang 87 20 0 04 Feb 2024
A Closer Look at the Limitations of Instruction Tuning Sreyan Ghosh Chandra Kiran Reddy Evuru Sonal Kumar Reddy Evuru Deepali Aneja Zeyu Jin R. Duraiswami Dinesh Manocha ALM 75 28 0 03 Feb 2024
Next-Generation Simulation Illuminates Scientific Problems of Organised Complexity Cheng Wang Chuwen Wang Wang Zhang Shirong Zeng Yu Zhao Ronghui Ning Changjun Jiang 41 0 0 18 Jan 2024
Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classification Iain J. Cruickshank Lynnette Hui Xian Ng 24 9 0 24 Sep 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 319 11,953 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 382 8,495 0 28 Jan 2022
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,848 0 18 Apr 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 258 4,489 0 23 Jan 2020
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 175 3,510 0 10 Jun 2015