LoRA+: Efficient Low Rank Adaptation of Large Models

19 February 2024

Papers citing "LoRA+: Efficient Low Rank Adaptation of Large Models"

50 / 115 papers shown

Title
MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning Xujia Wang Haiyan Zhao Shuo Wang Hanqing Wang Zhiyuan Liu MoMe MoE 37 0 0 30 Oct 2024
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization Jui-Nan Yen Si Si Zhao Meng Felix X. Yu Sai Surya Duvvuri Inderjit Dhillon Cho-Jui Hsieh Sanjiv Kumar 27 3 0 27 Oct 2024
GeoLoRA: Geometric integration for parameter efficient fine-tuning Steffen Schotthöfer Emanuele Zangrando Gianluca Ceruti Francesco Tudisco J. Kusch AI4CE 31 1 0 24 Oct 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies Liwen Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 49 3 0 24 Oct 2024
Learning Mathematical Rules with Large Language Models Antoine Gorceix Bastien Le Chenadec Ahmad Rammal N. Vadori Manuela Veloso 23 1 0 22 Oct 2024
Can Large Language Models Act as Ensembler for Multi-GNNs? Hanqi Duan Yao Cheng Jianxiang Yu Xiang Li AI4CE 35 0 0 22 Oct 2024
Fine-Tuning LLMs for Reliable Medical Question-Answering Services Ali Anaissi Ali Braytee Junaid Akram LM&MA AI4MH 33 2 0 21 Oct 2024
Learning to Route LLMs with Confidence Tokens Yu-Neng Chuang Helen Zhou Prathusha Kameswara Sarma Parikshit Gopalan John Boccio Sara Bolouki Xia Hu 35 7 0 17 Oct 2024
ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flaws Hai Huang Randall Balestriero 35 0 0 13 Oct 2024
MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning Yaming Yang Dilxat Muhtar Yelong Shen Yuefeng Zhan Jianfeng Liu ... Denvy Deng Feng Sun Qi Zhang Weizhu Chen Yunhai Tong MoE MoMe 46 2 0 12 Oct 2024
Parameter-Efficient Fine-Tuning of State Space Models Kevin Galim Wonjun Kang Yuchen Zeng H. Koo Kangwook Lee 31 4 0 11 Oct 2024
One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation Fabian Paischer Lukas Hauzenberger Thomas Schmied Benedikt Alkin Marc Peter Deisenroth Sepp Hochreiter 37 4 0 09 Oct 2024
Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models Theo Putterman Derek Lim Yoav Gelberg Stefanie Jegelka Haggai Maron AI4CE 43 5 0 05 Oct 2024
Hyperbolic Fine-tuning for Large Language Models Menglin Yang Aosong Feng Bo Xiong Jihong Liu Irwin King Rex Ying 23 1 0 05 Oct 2024
LoRTA: Low Rank Tensor Adaptation of Large Language Models Ignacio Hounie Charilaos I. Kanatsoulis Arnuv Tandon Alejandro Ribeiro 36 0 0 05 Oct 2024
AutoLoRA: AutoGuidance Meets Low-Rank Adaptation for Diffusion Models Artur Kasymov Marcin Sendera Michał Stypułkowski Maciej Ziȩba Przemysław Spurek 33 1 0 04 Oct 2024
Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization Xinhao Yao Hongjin Qian Xiaolin Hu Gengze Xu Wei Liu Jian Luan Bin Wang Yong-Jin Liu 48 0 0 03 Oct 2024
Selective Aggregation for Low-Rank Adaptation in Federated Learning Pengxin Guo Shuang Zeng Y. Wang Huijie Fan Feifei Wang Liangqiong Qu FedML 44 10 0 02 Oct 2024
LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models Haolin Li Yuhang Zhou Ziheng Zhao Siyuan Du Jiangchao Yao Weidi Xie Ya Zhang Yanfeng Wang 31 1 0 29 Sep 2024
Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape Tao Li Zhengbao He Yujun Li Yasheng Wang Lifeng Shang X. Huang 53 0 0 22 Sep 2024
Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy Bojian Li Bo Liu Jinghua Yue F. Zhou Fugen Zhou MedIm MDE 53 2 0 12 Sep 2024
Fast Forwarding Low-Rank Training Adir Rahamim Naomi Saphra Sara Kangaslahti Yonatan Belinkov 34 0 0 06 Sep 2024
EELE: Exploring Efficient and Extensible LoRA Integration in Emotional Text-to-Speech Xin Qi Ruibo Fu Zhengqi Wen Jianhua Tao Shuchen Shi ... Yuankun Xie Yukun Liu Guanjun Li Xuefei Liu Yongwei Li 43 1 0 20 Aug 2024
NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models Cheng Lin Lujun Li Dezhi Li Jie Zou Wei Xue Yike Guo AI4TS 46 4 0 18 Aug 2024
KIND: Knowledge Integration and Diversion for Training Decomposable Models Yucheng Xie Fu Feng Jing Wang Jing Wang Yong Rui Xin Geng VLM 36 3 0 14 Aug 2024
Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation Jingjing Xie Yuxin Zhang Mingbao Lin Liujuan Cao Rongrong Ji MQ 41 3 0 07 Aug 2024
Parameter-Efficient Fine-Tuning via Circular Convolution Aochuan Chen Jiashun Cheng Zijing Liu Ziqi Gao Fugee Tsung Yu Li Jia Li 56 2 0 27 Jul 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 61 16 0 25 Jul 2024
Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance Ao Shen Qiang Wang Zhiquan Lai Xionglve Li Dongsheng Li ALM MQ 35 1 0 24 Jul 2024
Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters Kartikeya Bhardwaj N. Pandey Sweta Priyadarshi Viswanath Ganapathy Rafael Esteves ... Paul N. Whatmough Risheek Garrepalli M. V. Baalen Harris Teague Markus Nagel MoMe 33 0 0 22 Jul 2024
Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approach Yuzhu Mao Siqi Ping Zihao Zhao Yang Liu Wenbo Ding 37 1 0 16 Jul 2024
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients Ajay Jaiswal Lu Yin Zhenyu (Allen) Zhang Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang 38 14 0 15 Jul 2024
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients Zhenyu (Allen) Zhang Ajay Jaiswal L. Yin Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang VLM 33 16 0 11 Jul 2024
RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization Xijie Huang Zechun Liu Shih-yang Liu Kwang-Ting Cheng MQ 43 7 0 10 Jul 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 54 25 0 08 Jul 2024
LoRA-GA: Low-Rank Adaptation with Gradient Approximation Shaowen Wang Linxi Yu Jian Li ALM AI4CE 26 30 0 06 Jul 2024
Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning Siwei Li Yifan Yang Yifei Shen Fangyun Wei Zongqing Lu L. Qiu Yuqing Yang AI4CE 43 1 0 01 Jul 2024
Octo-planner: On-device Language Model for Planner-Action Agents Wei Chen Zhiyuan Li Zhen Guo Yikang Shen LM&Ro LLMAG 31 4 0 26 Jun 2024
MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation Yusheng Liao Shuyang Jiang Yanfeng Wang Yu Wang 52 2 0 25 Jun 2024
Geneverse: A collection of Open-source Multimodal Large Language Models for Genomic and Proteomic Research Tianyu Liu Yijia Xiao Xiao Luo Hua Xu W. Zheng Hongyu Zhao 42 3 0 21 Jun 2024
On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion Chenghao Fan Zhenyi Lu Wei Wei Jie Tian Xiaoye Qu Dangyang Chen Yu Cheng MoMe 48 5 0 17 Jun 2024
Mixture-of-Subspaces in Low-Rank Adaptation Taiqiang Wu Jiahao Wang Zhe Zhao Ngai Wong 49 22 0 16 Jun 2024
Vertical LoRA: Dense Expectation-Maximization Interpretation of Transformers Zhuolin Fu 42 0 0 13 Jun 2024
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang Zeguan Xiao Shuo Wang Guanhua Chen Guanhua Chen 44 19 0 13 Jun 2024
The Impact of Initialization on LoRA Finetuning Dynamics Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 36 13 0 12 Jun 2024
Compute Better Spent: Replacing Dense Layers with Structured Matrices Shikai Qiu Andres Potapczynski Marc Finzi Micah Goldblum Andrew Gordon Wilson 40 11 0 10 Jun 2024
Low-Rank Adaption on Transformer-based Oriented Object Detector for Satellite Onboard Processing of Remote Sensing Images Xinyang Pu Feng Xu 32 3 0 04 Jun 2024
SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining Andi Han Jiaxiang Li Wei Huang Mingyi Hong Akiko Takeda Pratik Jawanpuria Bamdev Mishra 44 10 0 04 Jun 2024
OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models Kerim Büyükakyüz AI4CE 23 5 0 03 Jun 2024
QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation Zhuo Chen Rumen Dangovski Charlotte Loh Owen Dugan Di Luo Marin Soljacic MQ 30 8 0 31 May 2024