Fine-Tuning Language Models with Just Forward Passes

27 May 2023

Papers citing "Fine-Tuning Language Models with Just Forward Passes"

50 / 144 papers shown

Title
Parameter-Efficient Fine-Tuning via Circular Convolution Aochuan Chen Jiashun Cheng Zijing Liu Ziqi Gao Fugee Tsung Yu Li Jia Li 56 2 0 27 Jul 2024
Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration Tianyu Wang Sheng Li Bingyao Li Yuezhen Dai Ao Li Geng Yuan Yufei Ding Youtao Zhang Xulong Tang 40 0 0 18 Jul 2024
Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks Mingqing Xiao Qingyan Meng Zongpeng Zhang D.K. He Zhouchen Lin 40 0 0 17 Jul 2024
MINI-LLM: Memory-Efficient Structured Pruning for Large Language Models Hongrong Cheng Miao Zhang J. Q. Shi 57 2 0 16 Jul 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 42 43 0 09 Jul 2024
Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning Siwei Li Yifan Yang Yifei Shen Fangyun Wei Zongqing Lu L. Qiu Yuqing Yang AI4CE 43 1 0 01 Jul 2024
PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs Dan Peng Zhihui Fu Jun Wang 40 12 0 01 Jul 2024
Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs Enshu Liu Junyi Zhu Zinan Lin Xuefei Ning Matthew B. Blaschko Shengen Yan Guohao Dai Huazhong Yang Yu Wang MoE 62 5 0 01 Jul 2024
AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning Yifan Yang Kai Zhen Ershad Banijamal Athanasios Mouchtaris Zheng Zhang 42 8 0 26 Jun 2024
Adam-mini: Use Fewer Learning Rates To Gain More Yushun Zhang Congliang Chen Ziniu Li Tian Ding Chenwei Wu Yinyu Ye Zhi-Quan Luo Ruoyu Sun 46 37 0 24 Jun 2024
Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization Sungbin Shin Wonpyo Park Jaeho Lee Namhoon Lee 46 1 0 21 Jun 2024
Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization Qianli Shen Yezhen Wang Zhouhao Yang Xiang Li Haonan Wang Yang Zhang Jonathan Scarlett Zhanxing Zhu Kenji Kawaguchi AI4CE 72 4 0 20 Jun 2024
Synergizing Foundation Models and Federated Learning: A Survey Shenghui Li Fanghua Ye Meng Fang Jiaxu Zhao Yun-Hin Chan Edith C. -H. Ngai Thiemo Voigt AI4CE 57 5 0 18 Jun 2024
DIEKAE: Difference Injection for Efficient Knowledge Augmentation and Editing of Large Language Models Alessio Galatolo Meriem Beloucif Katie Winkle 41 0 0 15 Jun 2024
Minimizing Energy Costs in Deep Learning Model Training: The Gaussian Sampling Approach Challapalli Phanindra Revanth Sumohana S. Channappayya C Krishna Mohan 33 23 0 11 Jun 2024
Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity Wentao Guo Jikai Long Yimeng Zeng Zirui Liu Xinyu Yang ... Osbert Bastani Christopher De Sa Xiaodong Yu Beidi Chen Zhaozhuo Xu 34 14 0 05 Jun 2024
Why Larger Language Models Do In-context Learning Differently? Zhenmei Shi Junyi Wei Zhuoyan Xu Yingyu Liang 37 19 0 30 May 2024
Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient Hao Di Haishan Ye Yueling Zhang Xiangyu Chang Guang Dai Ivor W. Tsang 32 1 0 28 May 2024
Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective Akiyoshi Tomihari Issei Sato 30 4 0 27 May 2024
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization Zhe Li Bicheng Ying Zidong Liu Haibo Yang Haibo Yang FedML 59 3 0 24 May 2024
Thinking Forward: Memory-Efficient Federated Finetuning of Language Models Kunjal Panchal Nisarg Parikh Sunav Choudhary Lijun Zhang Yuriy Brun Hui Guan 61 3 0 24 May 2024
Efficient Multimodal Large Language Models: A Survey Yizhang Jin Jian Li Yexin Liu Tianjun Gu Kai Wu ... Xin Tan Zhenye Gan Yabiao Wang Chengjie Wang Lizhuang Ma LRM 47 45 0 17 May 2024
Binary Hypothesis Testing for Softmax Models and Leverage Score Models Yeqi Gao Yuzhou Gu Zhao Song 33 0 0 09 May 2024
Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning Jing Xu Jingzhao Zhang 39 7 0 04 May 2024
BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models Qi Luo Hengxu Yu Xiao Li 47 1 0 03 Apr 2024
Test-Time Model Adaptation with Only Forward Passes Shuaicheng Niu Chunyan Miao Guohao Chen Pengcheng Wu Peilin Zhao TTA 43 19 0 02 Apr 2024
Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better En-hao Liu Junyi Zhu Zinan Lin Xuefei Ning Shuaiqi Wang ... Sergey Yekhanin Guohao Dai Huazhong Yang Yu-Xiang Wang Yu Wang MoMe 57 4 0 02 Apr 2024
Heterogeneous Contrastive Learning for Foundation Models and Beyond Lecheng Zheng Baoyu Jing Zihao Li Hanghang Tong Jingrui He VLM 38 19 0 30 Mar 2024
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning Rui Pan Xiang Liu Shizhe Diao Renjie Pi Jipeng Zhang Chi Han Tong Zhang 46 37 0 26 Mar 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 318 0 21 Mar 2024
Debiased Noise Editing on Foundation Models for Fair Medical Image Classification Ruinan Jin Wenlong Deng Minghui Chen Xiaoxiao Li MedIm 45 1 0 10 Mar 2024
Privacy-preserving Fine-tuning of Large Language Models through Flatness Tiejin Chen Longchao Da Huixue Zhou Pingzhi Li Kaixiong Zhou Tianlong Chen Hua Wei 29 5 0 07 Mar 2024
Differentially Private Synthetic Data via Foundation Model APIs 2: Text Chulin Xie Zinan Lin A. Backurs Sivakanth Gopi Da Yu ... Haotian Jiang Huishuai Zhang Yin Tat Lee Bo-wen Li Sergey Yekhanin SyDa 63 34 0 04 Mar 2024
OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization Xiang Meng Shibal Ibrahim Kayhan Behdin Hussein Hazimeh Natalia Ponomareva Rahul Mazumder VLM 49 5 0 02 Mar 2024
A Survey of Large Language Models in Cybersecurity Gabriel de Jesus Coelho da Silva Carlos Becker Westphall 37 6 0 26 Feb 2024
Why Transformers Need Adam: A Hessian Perspective Yushun Zhang Congliang Chen Tian Ding Ziniu Li Ruoyu Sun Zhimin Luo 40 43 0 26 Feb 2024
Personalized Federated Instruction Tuning via Neural Architecture Search Peng Zhang Yingbo Zhou Ming Hu Junxian Feng Jiawen Weng Mingsong Chen FedML 30 4 0 26 Feb 2024
Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion Xuantong Liu Tianyang Hu Wei Cao Kenji Kawaguchi Yuan Yao DiffM 75 3 0 26 Feb 2024
Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning Yong Liu Zirui Zhu Chaoyu Gong Minhao Cheng Cho-Jui Hsieh Yang You MoE 42 16 0 24 Feb 2024
Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer Yanjun Zhao Sizhe Dang Haishan Ye Guang Dai Yi Qian Ivor W.Tsang 66 8 0 23 Feb 2024
A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu Ming Li Chongyang Tao Tao Shen Reynold Cheng Jinyang Li Can Xu Dacheng Tao Dinesh Manocha KELM VLM 44 102 0 20 Feb 2024
GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network Shuzhou Yuan Ercong Nie Michael Farber Helmut Schmid Hinrich Schütze 37 3 0 18 Feb 2024
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark Yihua Zhang Pingzhi Li Junyuan Hong Jiaxiang Li Yimeng Zhang ... Wotao Yin Mingyi Hong Zhangyang Wang Sijia Liu Tianlong Chen 28 45 0 18 Feb 2024
LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models Yifan Yang Jiajun Zhou Ngai Wong Zheng Zhang 23 7 0 18 Feb 2024
The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes Myeongseob Ko Feiyang Kang Weiyan Shi Ming Jin Zhou Yu Ruoxi Jia TDI 16 4 0 14 Feb 2024
Rethinking Machine Unlearning for Large Language Models Sijia Liu Yuanshun Yao Jinghan Jia Stephen Casper Nathalie Baracaldo ... Hang Li Kush R. Varshney Mohit Bansal Sanmi Koyejo Yang Liu AILaw MU 75 84 0 13 Feb 2024
Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning Zhicheng Liu Jian Lou Wenxuan Bao Yihan Hu Baochun Li Zhanyue Qin K. Ren 37 7 0 12 Feb 2024
On the Convergence of Zeroth-Order Federated Tuning for Large Language Models Zhenqing Ling Daoyuan Chen Liuyi Yao Yaliang Li Ying Shen FedML 47 12 0 08 Feb 2024
Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes Lucio Dery Steven Kolawole Jean-Francois Kagey Virginia Smith Graham Neubig Ameet Talwalkar 44 28 0 08 Feb 2024
The Fine-Grained Complexity of Gradient Computation for Training Large Language Models Josh Alman Zhao Song 27 12 0 07 Feb 2024