PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

3 April 2024

Papers citing "PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models"

30 / 80 papers shown

Title
QERA: an Analytical Framework for Quantization Error Reconstruction Cheng Zhang Jeffrey T. H. Wong Can Xiao George A. Constantinides Yiren Zhao MQ 52 2 0 08 Oct 2024
Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models Theo Putterman Derek Lim Yoav Gelberg Stefanie Jegelka Haggai Maron AI4CE 56 5 0 05 Oct 2024
FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models Yucheng Xie Fu Feng Ruixiao Shi Jing Wang Xin Geng AI4CE 41 2 0 28 Sep 2024
Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape Tao Li Zhengbao He Yujun Li Yasheng Wang Lifeng Shang Xiaolin Huang 53 0 0 22 Sep 2024
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models Yuncheng Yang Yulei Qin Tong Wu Zihan Xu Gang Li ... Yuchen Shi Ke Li Xing Sun Jie Yang Yun Gu ALM OffRL MoE 77 0 0 28 Aug 2024
SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models Yang Cao 52 2 0 21 Aug 2024
NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models Cheng Lin Lujun Li Dezhi Li Jie Zou Wei Xue Yike Guo AI4TS 51 4 0 18 Aug 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 75 21 0 25 Jul 2024
LoRA-PT: Low-Rank Adapting UNETR for Hippocampus Segmentation Using Principal Tensor Singular Values and Vectors Guanghua He Wangang Cheng Hancan Zhu Gaohang Yu 68 1 0 16 Jul 2024
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients Ajay Jaiswal Lu Yin Zhenyu Zhang Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang 43 14 0 15 Jul 2024
SLIP: Securing LLMs IP Using Weights Decomposition Yehonathan Refael Adam Hakim Lev Greenberg T. Aviv S. Lokam Ben Fishman Shachar Seidman 69 4 0 15 Jul 2024
A Survey on LoRA of Large Language Models Yuren Mao Yuhang Ge Yijiang Fan Wenyi Xu Yu Mi Zhonghao Hu Yunjun Gao ALM 63 30 0 08 Jul 2024
A Fingerprint for Large Language Models Zhiguang Yang Hanzhou Wu 67 9 0 01 Jul 2024
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead Rickard Brüel-Gabrielsson Jiacheng Zhu Onkar Bhardwaj Leshem Choshen Kristjan Greenewald Mikhail Yurochkin Justin Solomon 82 6 0 17 Jun 2024
The Impact of Initialization on LoRA Finetuning Dynamics Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 48 16 0 12 Jun 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou Shuaiwen Leon Song Jianlong Wu Liqiang Nie Guohao Li 48 9 0 07 Jun 2024
SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors Vijay Lingam Atula Tejaswi Aditya Vavre Aneesh Shetty Gautham Krishna Gudur Joydeep Ghosh Alexandros G. Dimakis Eunsol Choi Aleksandar Bojchevski Sujay Sanghavi 54 14 0 30 May 2024
LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters Klaudia Bałazy Mohammadreza Banaei Karl Aberer Jacek Tabor 48 22 0 27 May 2024
Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation Shen Yuan Haotian Liu Hongteng Xu 56 3 0 24 May 2024
Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks Jialin Zhao Yingtao Zhang Xinghang Li Huaping Liu C. Cannistraci 38 1 0 24 May 2024
LoRA Learns Less and Forgets Less D. Biderman Jose Javier Gonzalez Ortiz Jacob P. Portes Mansheej Paul Philip Greengard ... Sam Havens Vitaliy Chiley Jonathan Frankle Cody Blakeney John P. Cunningham CLL 69 120 0 15 May 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 154 336 0 21 Mar 2024
Gemma: Open Models Based on Gemini Research and Technology Gemma Team Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju ... Armand Joulin Noah Fiedel Evan Senter Alek Andreev Kathleen Kenealy VLM LLMAG 133 453 0 13 Mar 2024
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts Akari Asai Mohammadreza Salehi Matthew E. Peters Hannaneh Hajishirzi 134 100 0 24 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 459 12,359 0 04 Mar 2022
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer Tu Vu Brian Lester Noah Constant Rami Al-Rfou Daniel Cer VLM LRM 141 279 0 15 Oct 2021
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning Runxin Xu Fuli Luo Zhiyuan Zhang Chuanqi Tan Baobao Chang Songfang Huang Fei Huang LRM 153 180 0 13 Sep 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 298 3,917 0 18 Apr 2021
WARP: Word-level Adversarial ReProgramming Karen Hambardzumyan Hrant Khachatrian Jonathan May AAML 260 343 0 01 Jan 2021
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 316 7,044 0 20 Apr 2018