DoRA: Weight-Decomposed Low-Rank Adaptation

14 February 2024

Papers citing "DoRA: Weight-Decomposed Low-Rank Adaptation"

45 / 245 papers shown

Title
MemControl: Mitigating Memorization in Diffusion Models via Automated Parameter Selection Raman Dutt Pedro Sanchez Ondrej Bohdal Sotirios A. Tsaftaris Timothy M. Hospedales MedIm 36 0 0 29 May 2024
Low-rank finetuning for LLMs: A fairness perspective Saswat Das Marco Romanelli Cuong Tran Zarreen Reza B. Kailkhura Ferdinando Fioretto 40 1 0 28 May 2024
OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning Pengxiang Li Lu Yin Xiaowei Gao Shiwei Liu 34 7 0 28 May 2024
LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters Klaudia Bałazy Mohammadreza Banaei Karl Aberer Jacek Tabor 43 19 0 27 May 2024
DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution Yulong Mao Kaiyu Huang Changhao Guan Ganglin Bao Fengran Mo Jinan Xu 37 11 0 27 May 2024
On Fairness of Low-Rank Adaptation of Large Models Zhoujie Ding Ken Ziyu Liu Pura Peetathawatchai Berivan Isik Sanmi Koyejo 48 4 0 27 May 2024
Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation Shen Yuan Haotian Liu Hongteng Xu 44 2 0 24 May 2024
Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation Abhinav C. P. Jain Swarat Chaudhuri Thomas W. Reps Christopher M. Jermaine 28 1 0 24 May 2024
Sparse Matrix in Large Language Model Fine-tuning Haoze He Juncheng Billy Li Xuan Jiang Heather Miller MoE 27 3 0 24 May 2024
RE-Adapt: Reverse Engineered Adaptation of Large Language Models William Fleshman Benjamin Van Durme VLM 29 3 0 23 May 2024
Bitune: Bidirectional Instruction-Tuning D. J. Kopiczko Tijmen Blankevoort Yuki Markus Asano 32 2 0 23 May 2024
LoRA-Ensemble: Efficient Uncertainty Modelling for Self-attention Networks Michelle Halbheer Dominik J. Mühlematter Alexander Becker Dominik Narnhofer Helge Aasen Konrad Schindler Mehmet Özgür Türkoglu UQCV 38 1 0 23 May 2024
MiniCache: KV Cache Compression in Depth Dimension for Large Language Models Akide Liu Jing Liu Zizheng Pan Yefei He Gholamreza Haffari Bohan Zhuang MQ 35 30 0 23 May 2024
Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning Chongjie Si Xuehui Wang Xue Yang Zhengqin Xu Qingyun Li Jifeng Dai Yu Qiao Xiaokang Yang Wei Shen 31 8 0 23 May 2024
CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models Huiwen Wu Xiaohan Li Deyi Zhang Xiaogang Xu Xiaogang Xu Puning Zhao Zhe Liu FedML 37 0 0 22 May 2024
ReALLM: A general framework for LLM compression and fine-tuning Louis Leconte Lisa Bedin Van Minh Nguyen Eric Moulines MQ 41 0 0 21 May 2024
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning Ting Jiang Shaohan Huang Shengyue Luo Zihan Zhang Haizhen Huang ... Weiwei Deng Feng Sun Qi Zhang Deqing Wang Fuzhen Zhuang 37 33 0 20 May 2024
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts Minghao Wu Jiahao Xu Yulin Yuan Gholamreza Haffari Longyue Wang Weihua Luo Kaifu Zhang LLMAG 119 22 0 20 May 2024
LoRA Learns Less and Forgets Less D. Biderman Jose Javier Gonzalez Ortiz Jacob P. Portes Mansheej Paul Philip Greengard ... Sam Havens Vitaliy Chiley Jonathan Frankle Cody Blakeney John P. Cunningham CLL 43 112 0 15 May 2024
What matters when building vision-language models? Hugo Laurençon Léo Tronchon Matthieu Cord Victor Sanh VLM 43 157 0 03 May 2024
MANTIS: Interleaved Multi-Image Instruction Tuning Dongfu Jiang Xuan He Huaye Zeng Cong Wei Max W.F. Ku Qian Liu Wenhu Chen VLM MLLM 33 103 0 02 May 2024
FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition Yuxuan Yan Qianqian Yang Shunpu Tang Zhiguo Shi 38 14 0 29 Apr 2024
Towards Incremental Learning in Large Language Models: A Critical Review M. Jovanovic Peter Voss ELM CLL KELM 37 5 0 28 Apr 2024
Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models Jiawei Chen Dingkang Yang Yue Jiang Mingcheng Li Jinjie Wei Xiaolu Hou Lihua Zhang 56 6 0 25 Apr 2024
OpenELM: An Efficient Language Model Family with Open Training and Inference Framework Sachin Mehta Mohammad Hossein Sekhavat Qingqing Cao Maxwell Horton Yanzi Jin ... Iman Mirzadeh Mahyar Najibi Dmitry Belenko Peter Zatloukal Mohammad Rastegari OSLM AIFin 38 50 0 22 Apr 2024
MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts Dengchun Li Yingzi Ma Naizheng Wang Zhengmao Ye Zhiyuan Cheng ... Yan Zhang Lei Duan Jie Zuo Cal Yang Mingjie Tang MoE 40 44 0 22 Apr 2024
Mixed Text Recognition with Efficient Parameter Fine-Tuning and Transformer Da Chang Yu Li 72 2 0 19 Apr 2024
From Data Deluge to Data Curation: A Filtering-WoRA Paradigm for Efficient Text-based Person Search Jintao Sun Zhedong Zheng Gangyi Ding Gangyi Ding 40 7 0 16 Apr 2024
Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers Diana-Nicoleta Grigore Mariana-Iuliana Georgescu J. A. Justo T. Johansen Andreea-Iuliana Ionescu Radu Tudor Ionescu 36 0 0 14 Apr 2024
MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts Yusheng Liao Shuyang Jiang Yu Wang Yanfeng Wang MoE 36 5 0 13 Apr 2024
LLoCO: Learning Long Contexts Offline Sijun Tan Xiuyu Li Shishir G. Patil Ziyang Wu Tianjun Zhang Kurt Keutzer Joseph E. Gonzalez Raluca A. Popa RALM OffRL LLMAG 46 6 0 11 Apr 2024
ReFT: Representation Finetuning for Language Models Zhengxuan Wu Aryaman Arora Zheng Wang Atticus Geiger Daniel Jurafsky Christopher D. Manning Christopher Potts OffRL 38 58 0 04 Apr 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 64 73 0 03 Apr 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 318 0 21 Mar 2024
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models Yaowei Zheng Richong Zhang Junhao Zhang Yanhan Ye Zheyan Luo Zhangchi Feng Yongqiang Ma 55 398 0 20 Mar 2024
Can LLMs' Tuning Methods Work in Medical Multimodal Domain? Jiawei Chen Yue Jiang Dingkang Yang Mingcheng Li Jinjie Wei Ziyun Qian Lihua Zhang LM&MA 27 9 0 11 Mar 2024
ApiQ: Finetuning of 2-Bit Quantized Large Language Model Baohao Liao Christian Herold Shahram Khadivi Christof Monz CLL MQ 47 12 0 07 Feb 2024
Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning Wenhan Xia Chengwei Qin Elad Hazan 60 55 0 08 Jan 2024
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization Prateek Yadav Leshem Choshen Colin Raffel Mohit Bansal 32 13 0 22 Nov 2023
Instruction Tuning with GPT-4 Baolin Peng Chunyuan Li Pengcheng He Michel Galley Jianfeng Gao SyDa ALM LM&MA 162 585 0 06 Apr 2023
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark Ashwin Kalyan ELM ReLM LRM 211 1,113 0 20 Sep 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong Guosheng Lin MLLM BDL VLM CLIP 392 4,154 0 28 Jan 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 398 8,559 0 28 Jan 2022
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,858 0 18 Apr 2021
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval Jie Lei Licheng Yu Tamara L. Berg Joey Tianyi Zhou 119 275 0 24 Jan 2020