DoRA: Weight-Decomposed Low-Rank Adaptation

14 February 2024

Papers citing "DoRA: Weight-Decomposed Low-Rank Adaptation"

50 / 245 papers shown

Title
KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation Rambod Azimi Rishav Rishav M. Teichmann Samira Ebrahimi Kahou ALM 31 0 0 28 Oct 2024
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization Jui-Nan Yen Si Si Zhao Meng Felix X. Yu Sai Surya Duvvuri Inderjit Dhillon Cho-Jui Hsieh Sanjiv Kumar 27 3 0 27 Oct 2024
On the Crucial Role of Initialization for Matrix Factorization Bingcong Li Liang Zhang Aryan Mokhtari Niao He 28 1 0 24 Oct 2024
WAFFLE: Multi-Modal Model for Automated Front-End Development Shanchao Liang Nan Jiang Shangshu Qian Lin Tan 19 0 0 24 Oct 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies Liwen Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 49 3 0 24 Oct 2024
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning Jingfan Zhang Yi Zhao Dan Chen Xing Tian Huanran Zheng Wei Zhu MoE 40 12 0 23 Oct 2024
PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context Maximilian Augustin Syed Shakib Sarwar Mostafa Elhoushi Sai Qian Zhang Yuecheng Li B. D. Salvo 28 0 0 23 Oct 2024
Cross-model Control: Improving Multiple Large Language Models in One-time Training Jiayi Wu Hao Sun Hengyi Cai Lixin Su S. Wang Dawei Yin Xiang Li Ming Gao MU 39 0 0 23 Oct 2024
LoRA-C: Parameter-Efficient Fine-Tuning of Robust CNN for IoT Devices Chuntao Ding Xu Cao Jianhang Xie Linlin Fan Shangguang Wang Zhichao Lu 39 1 0 22 Oct 2024
Pyramid Vector Quantization for LLMs Tycho F. A. van der Ouderaa Maximilian L. Croci Agrin Hilmkil James Hensman MQ 31 1 0 22 Oct 2024
DEAN: Deactivating the Coupled Neurons to Mitigate Fairness-Privacy Conflicts in Large Language Models Chen Qian Dongrui Liu Jie Zhang Yong Liu Jing Shao 40 1 0 22 Oct 2024
Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models Yuheng Lu Bingshuo Qian Caixia Yuan Huixing Jiang Xiaojie Wang CLL 34 0 0 22 Oct 2024
Fine-Tuning LLMs for Reliable Medical Question-Answering Services Ali Anaissi Ali Braytee Junaid Akram LM&MA AI4MH 33 2 0 21 Oct 2024
Towards Optimal Adapter Placement for Efficient Transfer Learning Aleksandra I. Nowak Otniel-Bogdan Mercea Anurag Arnab Jonas Pfeiffer Yann N. Dauphin Utku Evci 25 0 0 21 Oct 2024
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics Thomas Robert M. Safaryan Ionut-Vlad Modoranu Dan Alistarh ODL 36 2 0 21 Oct 2024
LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning Yiming Shi Jiwei Wei Yujia Wu Ran Ran Chengwei Sun Shiyuan He Yang Yang ALM 43 1 0 17 Oct 2024
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning Chuanyu Tang Yilong Chen Zhenyu Zhang Junyuan Shang Wenyuan Zhang Yong Huang Tingwen Liu MoE 26 0 0 17 Oct 2024
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models Sajjad Ghiasvand Yifan Yang Zhiyu Xue Mahnoosh Alizadeh Zheng Zhang Ramtin Pedarsani FedML 38 3 0 16 Oct 2024
Prompt Compression for Large Language Models: A Survey Zongqian Li Yinhong Liu Yixuan Su Nigel Collier MQ 52 10 0 16 Oct 2024
Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models Kai Yao P. Gao Lichun Li Yuan Zhao Xiaofeng Wang Wei Wang Jianke Zhu 26 1 0 15 Oct 2024
LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Models Hossein Abdi Mingfei Sun Andi Zhang Samuel Kaski Wei Pan 30 0 0 15 Oct 2024
DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model Yuqi Wang Ke Cheng Jiawei He Qitai Wang Hengchen Dai Yuntao Chen Fei Xia Zhaoxiang Zhang VGen 43 8 0 14 Oct 2024
BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation Peijia Qin Ruiyi Zhang Pengtao Xie 31 1 0 13 Oct 2024
ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flaws Hai Huang Randall Balestriero 35 0 0 13 Oct 2024
MoIN: Mixture of Introvert Experts to Upcycle an LLM Ajinkya Tejankar K. Navaneet Ujjawal Panchal Kossar Pourahmadi Hamed Pirsiavash MoE 29 0 0 13 Oct 2024
RepMatch: Quantifying Cross-Instance Similarities in Representation Space Mohammad Reza Modarres Sina Abbasi Mohammad Taher Pilehvar 29 0 0 12 Oct 2024
MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning Yaming Yang Dilxat Muhtar Yelong Shen Yuefeng Zhan Jianfeng Liu ... Denvy Deng Feng Sun Qi Zhang Weizhu Chen Yunhai Tong MoE MoMe 46 2 0 12 Oct 2024
Parameter-Efficient Fine-Tuning of State Space Models Kevin Galim Wonjun Kang Yuchen Zeng H. Koo Kangwook Lee 31 4 0 11 Oct 2024
Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models Adriana Fernandez-Lopez Shiwei Liu L. Yin Stavros Petridis Maja Pantic 29 0 0 10 Oct 2024
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture Jiayi Han Liang Du Hongwei Du Xiangguo Zhou Yiwen Wu Weibo Zheng Donghong Han CLL MoMe MoE 38 2 0 10 Oct 2024
One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation Fabian Paischer Lukas Hauzenberger Thomas Schmied Benedikt Alkin Marc Peter Deisenroth Sepp Hochreiter 37 4 0 09 Oct 2024
Parameter-Efficient Fine-Tuning via Selective Discrete Cosine Transform Yixian Shen Qi Bi Jia-Hong Huang Hongyi Zhu Anuj Pathania 36 1 0 09 Oct 2024
Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs Ruijia Niu D. Wu Rose Yu Yi Ma 33 1 0 09 Oct 2024
Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models Theo Putterman Derek Lim Yoav Gelberg Stefanie Jegelka Haggai Maron AI4CE 43 5 0 05 Oct 2024
Hyperbolic Fine-tuning for Large Language Models Menglin Yang Aosong Feng Bo Xiong Jihong Liu Irwin King Rex Ying 23 1 0 05 Oct 2024
Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization Zichen Miao Zhengyuan Yang Kevin Lin Ze Wang Zicheng Liu Lijuan Wang Qiang Qiu 48 3 0 04 Oct 2024
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization Xinhao Yao Hongjin Qian Xiaolin Hu Gengze Xu Wei Liu Jian Luan Bin Wang Yong-Jin Liu 48 0 0 03 Oct 2024
Selective Aggregation for Low-Rank Adaptation in Federated Learning Pengxin Guo Shuang Zeng Y. Wang Huijie Fan Feifei Wang Liangqiong Qu FedML 44 10 0 02 Oct 2024
FlashMask: Efficient and Rich Mask Extension of FlashAttention Guoxia Wang Jinle Zeng Xiyuan Xiao Siming Wu Jiabin Yang Lujing Zheng Zeyu Chen Jiang Bian Dianhai Yu Haifeng Wang 157 2 0 02 Oct 2024
House of Cards: Massive Weights in LLMs Jaehoon Oh Seungjun Shin Dokwan Oh 40 1 0 02 Oct 2024
LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models Haolin Li Yuhang Zhou Ziheng Zhao Siyuan Du Jiangchao Yao Weidi Xie Ya Zhang Yanfeng Wang 31 1 0 29 Sep 2024
FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models Yucheng Xie Fu Feng Ruixiao Shi Jing Wang Xin Geng AI4CE 36 2 0 28 Sep 2024
PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning Qibin Wang Xiaolin Hu Weikai Xu Wei Liu Jian Luan Bin Wang 33 1 0 25 Sep 2024
NanoVoice: Efficient Speaker-Adaptive Text-to-Speech for Multiple Speakers Nohil Park Heeseung Kim Che Hyun Lee Jooyoung Choi Jiheum Yeom Sungroh Yoon 27 2 0 24 Sep 2024
Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely Siyun Zhao Yuqing Yang Zilong Wang Zhiyuan He Luna Qiu Lili Qiu SyDa RALM 3DV 44 35 0 23 Sep 2024
Orthogonal Finetuning for Direct Preference Optimization Chenxu Yang Ruipeng Jia Naibin Gu Zheng-Shen Lin Siyuan Chen Chao Pang Weichong Yin Yu Sun Hua Wu Weiping Wang 37 0 0 23 Sep 2024
Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape Tao Li Zhengbao He Yujun Li Yasheng Wang Lifeng Shang X. Huang 53 0 0 22 Sep 2024
COMFORT: A Continual Fine-Tuning Framework for Foundation Models Targeted at Consumer Healthcare Chia-Hao Li N. Jha 23 1 0 14 Sep 2024
Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data Atilla Akkus Mingjie Li Junjie Chu Junjie Chu Michael Backes Sinem Sav Sinem Sav SILM SyDa 48 1 0 12 Sep 2024
Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach Meng Zhou Surajsinh Parmar Anubhav Bhatti LM&MA 30 0 0 09 Sep 2024