Language model compression with weighted low-rank factorization

30 June 2022

Papers citing "Language model compression with weighted low-rank factorization"

21 / 21 papers shown

Title
A3 : an Analytical Low-Rank Approximation Framework for Attention Jeffrey T. H. Wong Cheng Zhang Xinye Cao Pedro Gimenes George A. Constantinides Wayne Luk Yiren Zhao OffRL MQ 7 0 0 19 May 2025
Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition Zhiyuan Chen Keyi Li Yifan Jia Le Ye Yufei Ma DiffM 40 0 0 09 May 2025
LiteLMGuard: Seamless and Lightweight On-Device Prompt Filtering for Safeguarding Small Language Models against Quantization-induced Risks and Vulnerabilities Kalyan Nakka Jimmy Dani Ausmit Mondal Nitesh Saxena AAML 35 0 0 08 May 2025
Wanda++: Pruning Large Language Models via Regional Gradients Yifan Yang Kai Zhen Bhavana Ganesh Aram Galstyan Goeric Huybrechts ... S. Bodapati Nathan Susanj Zheng Zhang Jack FitzGerald Abhishek Kumar 66 0 0 06 Mar 2025
You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning Ayan Sengupta Siddhant Chaudhary Tanmoy Chakraborty 49 3 0 25 Jan 2025
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models Fan Wang Juyong Jiang Chansung Park Sunghun Kim Jing Tang 96 1 0 08 Dec 2024
MoDeGPT: Modular Decomposition for Large Language Model Compression Chi-Heng Lin Shangqian Gao James Seale Smith Abhishek Patel Shikhar Tuli Yilin Shen Hongxia Jin Yen-Chang Hsu 71 8 0 19 Aug 2024
Reweighted Solutions for Weighted Low Rank Approximation David P. Woodruff T. Yasuda 42 1 0 04 Jun 2024
SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression Xin Wang Yu Zheng Zhongwei Wan Mi Zhang MQ 57 44 0 12 Mar 2024
DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization Rahul Chand Yashoteja Prabhu Pratyush Kumar 20 3 0 20 Dec 2023
LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning Han Guo P. Greengard Eric P. Xing Yoon Kim MQ 38 44 0 20 Nov 2023
Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs Or Sharir Anima Anandkumar 32 0 0 27 Jul 2023
Low-Rank Prune-And-Factorize for Language Model Compression Siyu Ren Kenny Q. Zhu 16 9 0 25 Jun 2023
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation Yixiao Li Yifan Yu Qingru Zhang Chen Liang Pengcheng He Weizhu Chen Tuo Zhao 44 69 0 20 Jun 2023
Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation Zhao Song Mingquan Ye Junze Yin Licheng Zhang 34 7 0 07 Jun 2023
COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models Jinqi Xiao Miao Yin Yu Gong Xiao Zang Jian Ren Bo Yuan VLM ViT 45 9 0 26 May 2023
Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models Mohammadreza Banaei Klaudia Bałazy Artur Kasymov R. Lebret Jacek Tabor Karl Aberer OffRL 21 0 0 08 Feb 2023
Tensor Networks Meet Neural Networks: A Survey and Future Perspectives Maolin Wang Yu Pan Zenglin Xu Xiangli Yang Guangxi Li A. Cichocki Andrzej Cichocki 61 19 0 22 Jan 2023
Compressing Transformer-based self-supervised models for speech processing Tzu-Quan Lin Tsung-Huan Yang Chun-Yao Chang Kuang-Ming Chen Tzu-hsun Feng Hung-yi Lee Hao Tang 40 6 0 17 Nov 2022
Numerical Optimizations for Weighted Low-rank Estimation on Language Model Ting Hua Yen-Chang Hsu Felicity Wang Qiang Lou Yilin Shen Hongxia Jin 27 13 0 02 Nov 2022
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 299 6,996 0 20 Apr 2018