When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

27 February 2024

Papers citing "When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method"

50 / 76 papers shown

Title
Parallel Scaling Law for Language Models Mouxiang Chen Binyuan Hui Zeyu Cui Jiaxi Yang Dayiheng Liu Jianling Sun Junyang Lin Zhongxin Liu MoE LRM 37 0 0 15 May 2025
Position: Enough of Scaling LLMs! Lets Focus on Downscaling Ayan Sengupta Yash Goel Tanmoy Chakraborty 34 0 0 02 May 2025
LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection Xinyue Zeng Haohui Wang Junhong Lin Jun Wu Tyler Cody Dawei Zhou 100 0 0 01 May 2025
Reason Like a Radiologist: Chain-of-Thought and Reinforcement Learning for Verifiable Report Generation Peiyuan Jing Kinhei Lee Zhenxuan Zhang Huichi Zhou Zhengqing Yuan Zhifan Gao Lei Zhu G. Papanastasiou Yingying Fang Guang Yang MedIm OffRL LRM 65 0 0 25 Apr 2025
Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey Aoran Gan Hao Yu Kai Zhang Qi Liu Wenyu Yan Zhenya Huang Shiwei Tong Guoping Hu RALM 3DV 43 0 0 21 Apr 2025
Data-efficient LLM Fine-tuning for Code Generation Weijie Lv X. Xia Sheng-Jun Huang ALM SyDa 41 0 0 17 Apr 2025
Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs? Hansi Zeng Kai Hui Honglei Zhuang Zhen Qin Zhenrui Yue Hamed Zamani Dana Alon 35 0 0 16 Apr 2025
Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selection Haoming Wang Boyuan Yang Xiangyu Yin Wei Gao 33 0 0 15 Apr 2025
Training Small Reasoning LLMs with Cognitive Preference Alignment Wenrui Cai Chengyu Wang Junbing Yan Jun Huang Xiangzhong Fang LRM 26 1 0 14 Apr 2025
A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directions Chengyu Wang Taolin Zhang Richang Hong Jun Huang ReLM LRM 45 1 0 12 Apr 2025
LLM for Complex Reasoning Task: An Exploratory Study in Fermi Problems Z. Liu Carlos Rabat Villarreal Mostafa Rahgouy Amit Das Zheng Zhang Chang Ren Dongji Feng ReLM LRM 54 0 0 03 Apr 2025
Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems? Kai Yan Yufei Xu Zhengyin Du Xuesong Yao Zhilin Wang Xiaowen Guo Jiecao Chen ReLM ELM LRM 95 3 0 01 Apr 2025
Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agent Humza Nusrat Bing Luo Ryan Hall Joshua Kim H. Bagher-Ebadian Anthony Doemer B. Movsas Kundan Thind AI4CE 34 0 0 21 Mar 2025
Scaling Sparse and Dense Retrieval in Decoder-Only LLMs Hansi Zeng Julian Killingback Hamed Zamani RALM 78 1 0 24 Feb 2025
How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines Ayan Sengupta Yash Goel Tanmoy Chakraborty 50 0 0 17 Feb 2025
Factual Inconsistency in Data-to-Text Generation Scales Exponentially with LLM Size: A Statistical Validation Joy Mahapatra Soumyajit Roy Utpal Garain HILM ALM 85 0 0 17 Feb 2025
Refine Knowledge of Large Language Models via Adaptive Contrastive Learning Hai-Tao Zheng Haojing Huang Jiayi Kuang Yangning Li Shu Guo C. Qu Xiaoyu Tan Hai-Tao Zheng Ying Shen Philip S. Yu CLL 66 5 0 11 Feb 2025
From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning Yafu Li Zhilin Wang Tingchen Fu Ganqu Cui Sen Yang Yu Cheng 45 1 0 21 Jan 2025
The Scaling Law for LoRA Base on Mutual Information Upper Bound Jing Zhang Hui Gao Peng Zhang Shuzhen Sun Chang Yang Yuexian Hou 28 0 0 06 Jan 2025
Formal Mathematical Reasoning: A New Frontier in AI Kaiyu Yang Gabriel Poesia Jingxuan He Wenda Li Kristin Lauter Swarat Chaudhuri Dawn Song LRM AI4CE 82 21 0 20 Dec 2024
EvoWiki: Evaluating LLMs on Evolving Knowledge Wei Tang Yixin Cao Yang Deng Jiahao Ying Bo Wang ... Yuyue Zhao Qi Zhang Xuanjing Huang Yugang Jiang Yong Liao KELM 65 1 0 18 Dec 2024
OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones? Z. Chen Tingzhu Chen Wenjun Zhang Guangtao Zhai 96 3 0 02 Dec 2024
Unified Parameter-Efficient Unlearning for LLMs Chenlu Ding Jiancan Wu Yancheng Yuan Jinda Lu Kai Zhang Alex Su Xiang Wang Xiangnan He MU KELM 100 6 0 30 Nov 2024
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning Elita Lobo Chirag Agarwal Himabindu Lakkaraju LRM 72 5 0 22 Nov 2024
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization Hexuan Deng Wenxiang Jiao Xuebo Liu Min Zhang Zhaopeng Tu VLM 77 0 0 21 Nov 2024
FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training Philip Zmushko Aleksandr Beznosikov Martin Takáč Samuel Horváth 44 0 0 12 Nov 2024
CULL-MT: Compression Using Language and Layer pruning for Machine Translation Pedram Rostami M. Dousti 32 0 0 10 Nov 2024
Training Compute-Optimal Protein Language Models Xingyi Cheng Bo Chen Pan Li Jing Gong Jie Tang Le Song 84 13 0 04 Nov 2024
Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation Bohan Lyu Yadi Cao Duncan Watson-Parris Leon Bergen Taylor Berg-Kirkpatrick Rose Yu 61 3 0 01 Nov 2024
Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness Qi Zhang Yifei Wang Jingyi Cui Xiang Pan Qi Lei Stefanie Jegelka Yisen Wang AAML 34 1 0 27 Oct 2024
Compute-Constrained Data Selection Junjie Oscar Yin Alexander M. Rush 39 0 0 21 Oct 2024
Optimizing Large Language Models for Dynamic Constraints through Human-in-the-Loop Discriminators Timothy Wei Annabelle Miin Anastasia Miin 18 0 0 19 Oct 2024
Balancing Label Quantity and Quality for Scalable Elicitation Alex Troy Mallen Nora Belrose 34 1 0 17 Oct 2024
A Hitchhiker's Guide to Scaling Law Estimation Leshem Choshen Yang Zhang Jacob Andreas 41 6 0 15 Oct 2024
A Scalable Communication Protocol for Networks of Large Language Models Samuele Marro Emanuele La Malfa Jesse Wright Bernard Ghanem Nigel Shadbolt Michael Wooldridge Philip H. S. Torr GNN AIFin 43 8 0 14 Oct 2024
Speculative Coreset Selection for Task-Specific Fine-tuning Xiaoyu Zhang Juan Zhai Shiqing Ma Chao Shen Tianlin Li Weipeng Jiang Yang Liu 30 1 0 02 Oct 2024
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 54 4 0 30 Sep 2024
Revisiting the Superficial Alignment Hypothesis Mohit Raghavendra Vaskar Nath Sean Hendryx LRM 23 0 0 27 Sep 2024
CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair Mingjie Liu Yun-Da Tsai Wenfei Zhou Haoxing Ren SyDa 3DV 45 6 0 19 Sep 2024
Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning Essa Jan Nouar Aldahoul Moiz Ali Faizan Ahmad Fareed Zaffar Yasir Zaki 29 3 0 18 Sep 2024
Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison Judy Hanwen Shen Archit Sharma Jun Qin 42 4 0 15 Sep 2024
A Fused Large Language Model for Predicting Startup Success Abdurahman Maarouf Stefan Feuerriegel Nicolas Pröllochs 44 6 0 05 Sep 2024
An Empirical Study of Scaling Laws for Transfer Matthew Barnett 30 1 0 30 Aug 2024
Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation Vivek Iyer Bhavitvya Malik Pavel Stepachev Pinzhen Chen Barry Haddow Alexandra Birch ALM 31 3 0 23 Aug 2024
Fine-tuning multilingual language models in Twitter/X sentiment analysis: a study on Eastern-European V4 languages Tomás Filip Martin Pavlícek Petr Sosík 30 2 0 04 Aug 2024
Are Bigger Encoders Always Better in Vision Large Models? Bozhou Li Hao Liang Zimo Meng Wentao Zhang VLM 40 3 0 01 Aug 2024
Model Agnostic Hybrid Sharding For Heterogeneous Distributed Inference Claudio Angione Yue Zhao Harry Yang Ahmad Farhan Fielding Johnston James Buban Patrick Colangelo 42 1 0 29 Jul 2024
The Impact of LoRA Adapters for LLMs on Clinical NLP Classification Under Data Limitations Thanh-Dung Le T. Nguyen Vu Nguyen Ha 31 0 0 27 Jul 2024
Stay Tuned: An Empirical Study of the Impact of Hyperparameters on LLM Tuning in Real-World Applications Alon Halfon Shai Gretz Ofir Arviv Artem Spector Orith Toledo-Ronen Yoav Katz L. Ein-Dor Michal Shmueli-Scheuer Noam Slonim 31 2 0 25 Jul 2024
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients Zhenyu (Allen) Zhang Ajay Jaiswal L. Yin Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang VLM 33 16 0 11 Jul 2024