v1v2 (latest)

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models

16 October 2024

Papers citing "Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models"

50 / 53 papers shown

Title
Multi-model Ensemble Conformal Prediction in Dynamic Environments Erfan Hajihashemi Yanning Shen 61 2 0 06 Nov 2024
Selective Aggregation for Low-Rank Adaptation in Federated Learning Pengxin Guo Shuang Zeng Y. Wang Huijie Fan Feifei Wang Liangqiong Qu FedML 134 22 0 02 Oct 2024
FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations Ziyao Wang Zheyu Shen Yexiao He Guoheng Sun Hongyi Wang Lingjuan Lyu Ang Li 91 49 0 09 Sep 2024
AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning Yifan Yang Kai Zhen Ershad Banijamal Athanasios Mouchtaris Zheng Zhang 71 9 0 26 Jun 2024
Federated LoRA with Sparse Communication Kevin Kuo Arian Raje Kousik Rajesh Virginia Smith 98 11 0 07 Jun 2024
CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization Zi Yang Samridhi Choudhary Xinfeng Xie Cao Gao Siegfried Kunzmann Zheng Zhang VLM 128 10 0 23 May 2024
Robust Decentralized Learning with Local Updates and Gradient Tracking Sajjad Ghiasvand Amirhossein Reisizadeh Mahnoosh Alizadeh Ramtin Pedarsani 100 4 0 02 May 2024
FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition Yuxuan Yan Qianqian Yang Shunpu Tang Zhiguo Shi 84 18 0 29 Apr 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 263 398 0 21 Mar 2024
Improving LoRA in Privacy-preserving Federated Learning Youbang Sun Zitao Li Yaliang Li Bolin Ding 84 81 0 18 Mar 2024
LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models Yifan Yang Jiajun Zhou Ngai Wong Zheng Zhang 70 7 0 18 Feb 2024
DoRA: Weight-Decomposed Low-Rank Adaptation Shih-yang Liu Chien-Yi Wang Hongxu Yin Pavlo Molchanov Yu-Chiang Frank Wang Kwang-Ting Cheng Min-Hung Chen 129 418 0 14 Feb 2024
Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models Yae Jee Cho Luyang Liu Zheng Xu Aldi Fahrezi Gauri Joshi 71 67 0 12 Jan 2024
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization Tianshi Che Ji Liu Yang Zhou Jiaxiang Ren Jiwen Zhou Victor S. Sheng H. Dai Dejing Dou 81 55 0 23 Oct 2023
FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models Tao Fan Yan Kang Guoqiang Ma Weijing Chen Wenbin Wei Lixin Fan Qiang Yang 87 65 0 16 Oct 2023
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models Sara Babakniya A. Elkordy Yahya H. Ezzeldin Qingfeng Liu Kee-Bong Song Mostafa El-Khamy Salman Avestimehr 73 71 0 12 Aug 2023
Parameter-Efficient Fine-Tuning without Introducing New Latency Baohao Liao Yan Meng Christof Monz 53 56 0 26 May 2023
Federated Learning of Medical Concepts Embedding using BEHRT Ofir Ben Shoham Nadav Rappoport FedML OOD 86 12 0 22 May 2023
Efficient Secure Aggregation for Privacy-Preserving Federated Machine Learning R. Behnia Mohammadreza Ebrahimi Arman Riasi Sherman S. M. Chow B. Padmanabhan Thang Hoang 72 7 0 07 Apr 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,761 0 15 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.5K 13,490 0 27 Feb 2023
When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods Zhuo Zhang Yuanhang Yang Yong Dai Zhuang Li Zenglin Xu FedML 117 81 0 20 Dec 2022
Federated Multilingual Models for Medical Transcript Analysis Andre Manoel Mirian Hipolito Garcia Tal Baumel Shize Su Jialei Chen Dan Miller D. Karmon Robert Sim Dimitrios Dimitriadis 61 13 0 04 Nov 2022
SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters Shwai He Liang Ding Daize Dong Miao Zhang Dacheng Tao MoE 112 90 0 09 Oct 2022
FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learning Haodong Zhao Wei Du Fang Li Peixuan Li Gongshen Liu FedML 48 72 0 25 Aug 2022
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning Haokun Liu Derek Tam Mohammed Muqeeth Jay Mohta Tenghao Huang Joey Tianyi Zhou Colin Raffel 107 934 0 11 May 2022
Towards a Unified View of Parameter-Efficient Transfer Learning Junxian He Chunting Zhou Xuezhe Ma Taylor Berg-Kirkpatrick Graham Neubig AAML 146 953 0 08 Oct 2021
Opacus: User-Friendly Differential Privacy Library in PyTorch Ashkan Yousefpour I. Shilov Alexandre Sablayrolles Davide Testuggine Karthik Prasad ... Sayan Gosh Akash Bharadwaj Jessica Zhao Graham Cormode Ilya Mironov VLM 291 368 0 25 Sep 2021
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models Elad Ben-Zaken Shauli Ravfogel Yoav Goldberg 202 1,244 0 18 Jun 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 509 10,526 0 17 Jun 2021
FedScale: Benchmarking Model and System Performance of Federated Learning at Scale Fan Lai Yinwei Dai Sanjay Sri Vallabh Singapuram Jiachen Liu Xiangfeng Zhu H. Madhyastha Mosharaf Chowdhury FedML 109 204 0 24 May 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 620 4,099 0 18 Apr 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 252 4,305 0 01 Jan 2021
Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity Amirhossein Reisizadeh Isidoros Tziotis Hamed Hassani Aryan Mokhtari Ramtin Pedarsani FedML 218 102 0 28 Dec 2020
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown Basel Alomair Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 522 1,956 0 14 Dec 2020
Parameter-Efficient Transfer Learning with Diff Pruning Demi Guo Alexander M. Rush Yoon Kim 84 406 0 14 Dec 2020
DeBERTa: Decoding-enhanced BERT with Disentangled Attention Pengcheng He Xiaodong Liu Jianfeng Gao Weizhu Chen AAML 169 2,761 0 05 Jun 2020
Advances and Open Problems in Federated Learning Peter Kairouz H. B. McMahan Brendan Avent A. Bellet M. Bennis ... Zheng Xu Qiang Yang Felix X. Yu Han Yu Sen Zhao FedML AI4CE 283 6,307 0 10 Dec 2019
The Non-IID Data Quagmire of Decentralized Machine Learning Kevin Hsieh Amar Phanishayee O. Mutlu Phillip B. Gibbons 182 573 0 01 Oct 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 703 24,572 0 26 Jul 2019
The Value of Collaboration in Convex Machine Learning with Differential Privacy Nan Wu Farhad Farokhi David B. Smith M. Kâafar FedML 72 98 0 24 Jun 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems Alex Jinpeng Wang Yada Pruksachatkun Nikita Nangia Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 279 2,326 0 02 May 2019
Parameter-Efficient Transfer Learning for NLP N. Houlsby A. Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin de Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly 226 4,529 0 02 Feb 2019
Federated Optimization in Heterogeneous Networks Tian Li Anit Kumar Sahu Manzil Zaheer Maziar Sanjabi Ameet Talwalkar Virginia Smith FedML 192 5,247 0 14 Dec 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,324 0 11 Oct 2018
Know What You Don't Know: Unanswerable Questions for SQuAD Pranav Rajpurkar Robin Jia Percy Liang RALM ELM 292 2,854 0 11 Jun 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,201 0 20 Apr 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 814 132,725 0 12 Jun 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 526 4,497 0 18 Apr 2017
Deep Learning with Differential Privacy Martín Abadi Andy Chu Ian Goodfellow H. B. McMahan Ilya Mironov Kunal Talwar Li Zhang FedML SyDa 223 27 0 01 Jul 2016