PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

3 April 2024

Papers citing "PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models"

50 / 75 papers shown

Title
CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation Guang Yang Yu Zhou Xiang Chen Wei-Shi Zheng Xing Hu Xin Zhou David Lo Taolue Chen ALM LRM 7 0 0 26 May 2025
Optimization-Inspired Few-Shot Adaptation for Large Language Models Boyan Gao Xin Wang Yibo Yang David A. Clifton 5 0 0 25 May 2025
ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation Jian Liang Wenke Huang Xianda Guo Guancheng Wan Bo Du Mang Ye MoMe MoE 2 0 0 24 May 2025
RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models Yilang Zhang Bingcong Li G. Giannakis 5 0 0 24 May 2025
Understanding Differential Transformer Unchains Pretrained Self-Attentions Chaerin Kong Jiho Jang Nojun Kwak 21 0 0 22 May 2025
HOFT: Householder Orthogonal Fine-tuning Alejandro Moreno Arcas Albert Sanchis Jorge Civera Alfons Juan 24 0 0 22 May 2025
ABBA: Highly Expressive Hadamard Product Adaptation for Large Language Models Raghav Singhal Kaustubh Ponkshe Rohit Vartak Praneeth Vepakomma 22 0 0 20 May 2025
FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA Seanie Lee Sangwoo Park Dong Bok Lee Dominik Wagner Haebin Seong Tobias Bocklet Juho Lee Sung Ju Hwang FedML 34 0 0 19 May 2025
AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections Xin Yu Yujia Wang Jinghui Chen Lingzhou Xue 29 0 0 18 May 2025
SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization Haodong Yang Lei Wang Md Zakir Hossain 42 0 0 18 May 2025
LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades Yanan Li Fanxu Meng Muhan Zhang Shiai Zhu Shangguang Wang Mengwei Xu MoMe 19 0 0 17 May 2025
Memory-Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation Fei Wu Jia Hu Geyong Min Shiqiang Wang 36 0 0 16 May 2025
Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptation Nairouz Mrabah Nicolas Richet Ismail Ben Ayed Eric Granger BDL VLM 67 0 0 16 Apr 2025
RWKVTTS: Yet another TTS based on RWKV-7 Lin Yueyu Liu Xiao 64 0 0 04 Apr 2025
Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformations Chongjie Si Zhiyi Shi Xuehui Wang Yichen Xiao Xiaokang Yang Wei Shen AI4CE 87 0 0 01 Apr 2025
MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning Jiancheng Zhao Xingda Yu Zhen Yang MoE 66 1 0 27 Mar 2025
Low-resource Machine Translation for Code-switched Kazakh-Russian Language Pair Maksim Borisov Zhanibek Kozhirbayev Valentin Malykh 62 0 0 25 Mar 2025
Hiding Images in Diffusion Models by Editing Learned Score Functions Haoyu Chen Yunqiao Yang Nan Zhong Kede Ma DiffM 81 0 0 24 Mar 2025
DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation Massimo Bini Leander Girrbach Zeynep Akata 52 0 0 23 Mar 2025
SALT: Singular Value Adaptation with Low-Rank Transformation Abdelrahman Elsayed Sarim Hashmi Mohammed Elseiagy Hu Wang Mohammad Yaqub Ibrahim Almakky OOD 69 0 0 20 Mar 2025
A Foundational individual Mobility Prediction Model based on Open-Source Large Language Models Zhenlin Qin Leizhen Wang Francisco Camara Pereira Zhenlinag Ma 70 0 0 19 Mar 2025
RaSA: Rank-Sharing Low-Rank Adaptation Zhiwei He Zhaopeng Tu Xing Wang Xingyu Chen Zhaoxiang Wang Jiahao Xu Tian Liang Wenxiang Jiao Zizhuo Zhang Rui Wang ALM 106 2 0 16 Mar 2025
Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization Ziqing Xu Hancheng Min Lachlan Ewen MacDonald Jinqi Luo Salma Tarmoun Enrique Mallada Rene Vidal AI4CE 61 0 0 10 Mar 2025
Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model Wenke Huang Jian Liang Xianda Guo Yiyang Fang Guancheng Wan ... Bin Yang He Li Jiawei Shao Mang Ye Di Lin OffRL LRM MLLM KELM VLM 80 2 0 06 Mar 2025
NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models Yibo Zhong Haoxiang Jiang Lincan Li Ryumei Nakada Tianci Liu Linjun Zhang Huaxiu Yao Haoyu Wang 85 2 0 24 Feb 2025
LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits Zikai Zhou Qizheng Zhang Hermann Kumbong Kunle Olukotun MQ 395 0 0 12 Feb 2025
RandLoRA: Full-rank parameter-efficient fine-tuning of large models Paul Albert Frederic Z. Zhang Hemanth Saratchandran Cristian Rodriguez-Opazo Anton van den Hengel Ehsan Abbasnejad 134 2 0 03 Feb 2025
Parameter-Efficient Fine-Tuning for Foundation Models Dan Zhang Tao Feng Lilong Xue Yuandong Wang Yuxiao Dong J. Tang 64 9 0 23 Jan 2025
A Domain Adaptation Framework for Speech Recognition Systems with Only Synthetic data Minh Tran Yutong Pang Debjyoti Paul Laxmi Pandey Kevin Jiang Jinxi Guo Ke Li Shun Zhang Xinyu Zhang Xin Lei AI4CE 51 0 0 21 Jan 2025
tCURLoRA: Tensor CUR Decomposition Based Low-Rank Parameter Adaptation and Its Application in Medical Image Segmentation Guanghua He Wangang Cheng Hancan Zhu Xiaohao Cai Gaohang Yu 58 0 0 10 Jan 2025
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement Tianyu Zheng Ge Zhang Tianhao Shen Xueling Liu Bill Yuchen Lin Jie Fu Wenhu Chen Xiang Yue SyDa 91 113 0 08 Jan 2025
Spectral-Aware Low-Rank Adaptation for Speaker Verification Zhe Li Man-Wai Mak Mert Pilanci Hung-yi Lee Helen Meng 60 0 0 07 Jan 2025
GaLore $+$ : Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection Xutao Liao Shaohui Li Yuhui Xu Zhi Li Yebin Liu You He VLM 72 3 0 31 Dec 2024
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models Xiaolin Hu Xiang Cheng Peiyu Liu Wei Liu Jian Luan Bin Wang Y. Liu 67 1 0 31 Dec 2024
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models Fan Wang Juyong Jiang Chansung Park Sunghun Kim Jing Tang 115 1 0 08 Dec 2024
Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA Haodong Lu Chongyang Zhao Jason Xue Lina Yao Kristen Moore Dong Gong CLL KELM VLM 113 5 0 01 Dec 2024
Reassessing Layer Pruning in LLMs: New Insights and Methods Yao Lu Hao Cheng Yujie Fang Zeyu Wang Jiaheng Wei Dongwei Xu Qi Xuan Xiaoniu Yang Zhaowei Zhu 77 0 0 23 Nov 2024
MC-LLaVA: Multi-Concept Personalized Vision-Language Model Ruichuan An Sihan Yang Ming Lu Kai Zeng Yulin Luo ... Hao Liang Qi She Shanghang Zhang Wentao Zhang Wentao Zhang 107 7 0 18 Nov 2024
LoRA vs Full Fine-tuning: An Illusion of Equivalence Reece Shuttleworth Jacob Andreas Antonio Torralba Pratyusha Sharma 67 10 0 28 Oct 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 84 5 0 28 Oct 2024
An Efficient Watermarking Method for Latent Diffusion Models via Low-Rank Adaptation Dongdong Lin Yue Li B. Tondi Bin Li Mauro Barni WIGM 55 1 0 26 Oct 2024
On the Crucial Role of Initialization for Matrix Factorization Bingcong Li Liang Zhang Aryan Mokhtari Niao He 72 1 0 24 Oct 2024
Understanding Layer Significance in LLM Alignment Guangyuan Shi Zexin Lu Xiaoyu Dong Wenlong Zhang Xuanyu Zhang Yujie Feng Xiao-Ming Wu 60 3 0 23 Oct 2024
LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning Yiming Shi Jiwei Wei Yujia Wu Ran Ran Chengwei Sun Shiyuan He Yang Yang ALM 50 1 0 17 Oct 2024
QERA: an Analytical Framework for Quantization Error Reconstruction Cheng Zhang Jeffrey T. H. Wong Can Xiao George A. Constantinides Yiren Zhao MQ 52 2 0 08 Oct 2024
Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models Theo Putterman Derek Lim Yoav Gelberg Stefanie Jegelka Haggai Maron AI4CE 56 5 0 05 Oct 2024
FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models Yucheng Xie Fu Feng Ruixiao Shi Jing Wang Xin Geng AI4CE 41 2 0 28 Sep 2024
Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape Tao Li Zhengbao He Yujun Li Yasheng Wang Lifeng Shang Xiaolin Huang 53 0 0 22 Sep 2024
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models Yuncheng Yang Yulei Qin Tong Wu Zihan Xu Gang Li ... Yuchen Shi Ke Li Xing Sun Jie Yang Yun Gu ALM OffRL MoE 75 0 0 28 Aug 2024
SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models Yang Cao 52 2 0 21 Aug 2024