DoRA: Weight-Decomposed Low-Rank Adaptation

14 February 2024

Papers citing "DoRA: Weight-Decomposed Low-Rank Adaptation"

50 / 245 papers shown

Title
Fast Forwarding Low-Rank Training Adir Rahamim Naomi Saphra Sara Kangaslahti Yonatan Belinkov 34 0 0 06 Sep 2024
Multi-Modal Adapter for Vision-Language Models Dominykas Seputis Serghei Mihailov Soham Chatterjee Zehao Xiao VLM 36 1 0 03 Sep 2024
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models Yuncheng Yang Yulei Qin Tong Wu Zihan Xu Gang Li ... Yuchen Shi Ke Li Xing Sun Jie Yang Yun Gu ALM OffRL MoE 54 0 0 28 Aug 2024
GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs Maxim Zhelnin Viktor Moskvoretskii Egor Shvetsov Egor Venediktov Mariya Krylova Aleksandr Zuev Evgeny Burnaev 32 2 0 27 Aug 2024
Building and better understanding vision-language models: insights and future directions Hugo Laurençon Andrés Marafioti Victor Sanh Léo Tronchon VLM 42 61 0 22 Aug 2024
SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models Yang Cao 44 2 0 21 Aug 2024
NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models Cheng Lin Lujun Li Dezhi Li Jie Zou Wei Xue Yike Guo AI4TS 46 4 0 18 Aug 2024
KIND: Knowledge Integration and Diversion for Training Decomposable Models Yucheng Xie Fu Feng Jing Wang Jing Wang Yong Rui Xin Geng VLM 36 3 0 14 Aug 2024
Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers Marcus Buckmann Edward Hill 40 2 0 06 Aug 2024
SARA: Singular-Value Based Adaptive Low-Rank Adaption Jihao Gu Shuai Chen Zelin Wang Yibo Zhang Ping Gong 38 3 0 06 Aug 2024
Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement Le Yu Bowen Yu Haiyang Yu Fei Huang Yongbin Li MoMe 35 5 0 06 Aug 2024
A Federated Learning-Friendly Approach for Parameter-Efficient Fine-Tuning of SAM in 3D Segmentation Mothilal Asokan Difei Gao Joya Chen Mike Zheng Shou FedML MedIm 39 1 0 31 Jul 2024
Parameter-Efficient Fine-Tuning via Circular Convolution Aochuan Chen Jiashun Cheng Zijing Liu Ziqi Gao Fugee Tsung Yu Li Jia Li 56 2 0 27 Jul 2024
Fine-Tuning Large Language Models for Stock Return Prediction Using Newsflow Tian Guo E. Hauptmann AIFin 41 3 0 25 Jul 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 61 16 0 25 Jul 2024
Structure-aware Domain Knowledge Injection for Large Language Models Kai-Chun Liu Ze Chen Zhihang Fu Rongxin Jiang Fan Zhou Yao-Shen Chen Yue-bo Wu Yue Wu Jieping Ye 55 1 0 23 Jul 2024
LoRA-PT: Low-Rank Adapting UNETR for Hippocampus Segmentation Using Principal Tensor Singular Values and Vectors Guanghua He Wangang Cheng Hancan Zhu Gaohang Yu 45 1 0 16 Jul 2024
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients Ajay Jaiswal Lu Yin Zhenyu (Allen) Zhang Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang 38 14 0 15 Jul 2024
Low-Rank Interconnected Adaptation Across Layers Yibo Zhong Yao Zhou OffRL MoE 48 1 0 13 Jul 2024
Still-Moving: Customized Video Generation without Customized Video Data Hila Chefer Shiran Zada Roni Paiss Ariel Ephrat Omer Tov Michael Rubinstein Lior Wolf Tali Dekel T. Michaeli Inbar Mosseri DiffM VGen 31 20 0 11 Jul 2024
Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Cross-Regularization Jinlong Li Zequn Jie Elisa Ricci Lin Ma N. Sebe VLM 39 1 0 11 Jul 2024
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients Zhenyu (Allen) Zhang Ajay Jaiswal L. Yin Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang VLM 33 16 0 11 Jul 2024
RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization Xijie Huang Zechun Liu Shih-yang Liu Kwang-Ting Cheng MQ 43 7 0 10 Jul 2024
Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences Nikolaos Dimitriadis Pascal Frossard F. Fleuret MoE 67 6 0 10 Jul 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 54 25 0 08 Jul 2024
See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition Chongjie Si Xiaokang Yang Wei Shen 45 5 0 07 Jul 2024
SBoRA: Low-Rank Adaptation with Regional Weight Updates L. Po Yuyang Liu Haoxuan Wu Tianqi Zhang W. Yu Zeyu Jiang Kun Li 38 1 0 07 Jul 2024
PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuning Jiaru Zou Mengyu Zhou Tao Li Shi Han Dongmei Zhang 51 6 0 02 Jul 2024
To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Models Bozhong Tian Xiaozhuan Liang Siyuan Cheng Qingbin Liu Mengru Wang Dianbo Sui Xi Chen Huajun Chen Ningyu Zhang MU 35 6 0 02 Jul 2024
Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning Siwei Li Yifan Yang Yifei Shen Fangyun Wei Zongqing Lu L. Qiu Yuqing Yang AI4CE 43 1 0 01 Jul 2024
ArzEn-LLM: Code-Switched Egyptian Arabic-English Translation and Speech Recognition Using LLMs Ahmed Heakl Youssef Zaghloul Mennatullah Ali Rania Hossam Walid Gomaa 19 2 0 26 Jun 2024
Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning Arijit Sehanobish Avinava Dubey Krzysztof Choromanski Somnath Basu Roy Chowdhury Deepali Jain Vikas Sindhwani Snigdha Chaturvedi ALM 43 1 0 25 Jun 2024
MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation Yusheng Liao Shuyang Jiang Yanfeng Wang Yu Wang 52 2 0 25 Jun 2024
Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs Ashwinee Panda Berivan Isik Xiangyu Qi Sanmi Koyejo Tsachy Weissman Prateek Mittal MoMe 45 13 0 24 Jun 2024
RE-AdaptIR: Improving Information Retrieval through Reverse Engineered Adaptation William Fleshman Benjamin Van Durme 48 0 0 20 Jun 2024
LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation Seyedarmin Azizi Souvik Kundu Massoud Pedram 32 7 0 18 Jun 2024
RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning Haoyu Wang Tianci Liu Ruirui Li Monica Cheng Tuo Zhao Jing Gao 29 7 0 16 Jun 2024
Mixture-of-Subspaces in Low-Rank Adaptation Taiqiang Wu Jiahao Wang Zhe Zhao Ngai Wong 49 22 0 16 Jun 2024
Group and Shuffle: Efficient Structured Orthogonal Parametrization Mikhail Gorbunov Nikolay Yudin Vera Soboleva Aibek Alanov Alexey Naumov Maxim Rakhuba 50 1 0 14 Jun 2024
FouRA: Fourier Low Rank Adaptation Shubhankar Borse Shreya Kadambi N. Pandey Kartikeya Bhardwaj Viswanath Ganapathy Sweta Priyadarshi Risheek Garrepalli Rafael Esteves Munawar Hayat Fatih Porikli 42 6 0 13 Jun 2024
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang Zeguan Xiao Shuo Wang Guanhua Chen Guanhua Chen 44 19 0 13 Jun 2024
The Impact of Initialization on LoRA Finetuning Dynamics Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 36 11 0 12 Jun 2024
Are Large Language Models Actually Good at Text Style Transfer? Sourabrata Mukherjee Atul Kr. Ojha Ondrej Dusek 29 11 0 09 Jun 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou Shuaiwen Leon Song Jianlong Wu Liqiang Nie Guohao Li 45 6 0 07 Jun 2024
Choice of PEFT Technique in Continual Learning: Prompt Tuning is Not All You Need Martin Wistuba Prabhu Teja Sivaprasad Lukas Balles Giovanni Zappella 37 0 0 05 Jun 2024
Low-Rank Adaption on Transformer-based Oriented Object Detector for Satellite Onboard Processing of Remote Sensing Images Xinyang Pu Feng Xu 32 3 0 04 Jun 2024
SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining Andi Han Jiaxiang Li Wei Huang Mingyi Hong Akiko Takeda Pratik Jawanpuria Bamdev Mishra 44 10 0 04 Jun 2024
QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation Zhuo Chen Rumen Dangovski Charlotte Loh Owen Dugan Di Luo Marin Soljacic MQ 30 8 0 31 May 2024
Visual Perception by Large Language Model's Weights Feipeng Ma Hongwei Xue Guangting Wang Yizhou Zhou Fengyun Rao Shilin Yan Yueyi Zhang Siying Wu Mike Zheng Shou Xiaoyan Sun VLM 25 5 0 30 May 2024
TAIA: Large Language Models are Out-of-Distribution Data Learners Shuyang Jiang Yusheng Liao Ya Zhang Yu Wang Yanfeng Wang 29 3 0 30 May 2024