Training Neural Networks with Fixed Sparse Masks

18 November 2021

Papers citing "Training Neural Networks with Fixed Sparse Masks"

50 / 143 papers shown

Title
SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining Andi Han Jiaxiang Li Wei Huang Mingyi Hong Akiko Takeda Pratik Jawanpuria Bamdev Mishra 41 10 0 04 Jun 2024
Sparsity-Accelerated Training for Large Language Models Da Ma Lu Chen Pengyu Wang Hongshen Xu Hanqi Li Liangtai Sun Su Zhu Shuai Fan Kai Yu LRM 33 0 0 03 Jun 2024
Lifelong Learning Using a Dynamically Growing Tree of Sub-networks for Domain Generalization in Video Object Segmentation Islam I. Osman Mohamed S. Shehata 35 0 0 29 May 2024
RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter Meng Cao Haoran Tang Jinfa Huang Peng Jin Can Zhang Ruyang Liu Long Chen Xiaodan Liang Li-ming Yuan Ge Li 98 11 0 29 May 2024
Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models Chia-Yi Hsu Yu-Lin Tsai Chih-Hsun Lin Pin-Yu Chen Chia-Mu Yu Chun-ying Huang 49 32 0 27 May 2024
Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation Shen Yuan Haotian Liu Hongteng Xu 44 2 0 24 May 2024
TriLoRA: Integrating SVD for Advanced Style Personalization in Text-to-Image Generation Chengcheng Feng Mu He Qiuyu Tian Haojie Yin Xiaofang Zhao Hongwei Tang Xingqiang Wei DiffM 30 3 0 18 May 2024
Pruning as a Domain-specific LLM Extractor Nan Zhang Yanchi Liu Xujiang Zhao Wei Cheng Runxue Bao Rui Zhang Prasenjit Mitra Haifeng Chen 26 9 0 10 May 2024
Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning Jing Xu Jingzhao Zhang 39 7 0 04 May 2024
AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees William Fleshman Aleem Khan Marc Marone Benjamin Van Durme CLL KELM 55 3 0 12 Apr 2024
Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models Zihan Fang Zheng Lin Zhe Chen Xianhao Chen Yue Gao Yuguang Fang 54 35 0 09 Apr 2024
Facial Affective Behavior Analysis with Instruction Tuning Yifan Li Anh Dao Wentao Bao Zhen Tan Tianlong Chen Huan Liu Yu Kong CVBM 60 15 0 07 Apr 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 64 73 0 03 Apr 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 310 0 21 Mar 2024
Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model Haoyun Xu Runzhe Zhan Derek F. Wong Lidia S. Chao 29 3 0 18 Mar 2024
Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generation Likun Li Haoqi Zeng Changpeng Yang Haozhe Jia Di Xu DiffM 34 4 0 12 Mar 2024
Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy? Nader Asadi Mahdi Beitollahi Yasser H. Khalil Yinchuan Li Guojun Zhang Xi Chen MoMe 37 8 0 23 Feb 2024
Modularized Networks for Few-shot Hateful Meme Detection Rui Cao Roy Ka-Wei Lee Jing Jiang 35 4 0 19 Feb 2024
Dynamic Layer Tying for Parameter-Efficient Transformers Tamir David Hay Lior Wolf 25 3 0 23 Jan 2024
APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference Bowen Zhao Hannaneh Hajishirzi Qingqing Cao 23 17 0 22 Jan 2024
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation Nadav Benedek Lior Wolf 26 5 0 20 Jan 2024
PersianMind: A Cross-Lingual Persian-English Large Language Model Pedram Rostami Ali Salemi M. Dousti CLL LRM 24 5 0 12 Jan 2024
RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation Mahdi Nikdan Soroush Tabesh Elvir Crnčević Dan Alistarh 8 27 0 09 Jan 2024
The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny? T. Besiroglu S. Bergerson Amelia Michael Lennart Heim Xueyun Luo Neil Thompson 22 11 0 04 Jan 2024
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment Lingling Xu Haoran Xie S. J. Qin Xiaohui Tao F. Wang 49 135 0 19 Dec 2023
Distributed Inference and Fine-tuning of Large Language Models Over The Internet Alexander Borzunov Max Ryabinin Artem Chumachenko Dmitry Baranchuk Tim Dettmers Younes Belkada Pavel Samygin Colin Raffel MoE ALM 15 39 0 13 Dec 2023
Batched Low-Rank Adaptation of Foundation Models Yeming Wen Swarat Chaudhuri OffRL 21 19 0 09 Dec 2023
Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts Jialin Wu Xia Hu Yaqing Wang Bo Pang Radu Soricut MoE 19 14 0 01 Dec 2023
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization Prateek Yadav Leshem Choshen Colin Raffel Mohit Bansal 32 13 0 22 Nov 2023
LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning Han Guo P. Greengard Eric P. Xing Yoon Kim MQ 36 43 0 20 Nov 2023
Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization Weiyang Liu Zeju Qiu Yao Feng Yuliang Xiu Yuxuan Xue ... Songyou Peng Yandong Wen Michael J. Black Adrian Weller Bernhard Schölkopf 50 57 0 10 Nov 2023
Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning Sarkar Snigdha Sarathi Das Ranran Haoran Zhang Peng Shi Wenpeng Yin Rui Zhang 115 14 0 07 Nov 2023
X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity Taejun Yun Jinhyeon Kim Deokyeong Kang Seong Hoon Lim Jihoon Kim Taeuk Kim 31 0 0 26 Oct 2023
On Surgical Fine-tuning for Language Encoders Abhilasha Lodha Gayatri Belapurkar Saloni Chalkapurkar Yuanming Tao Reshmi Ghosh Samyadeep Basu Dmitrii Petrov Soundararajan Srinivasan 20 3 0 25 Oct 2023
MACP: Efficient Model Adaptation for Cooperative Perception Yunsheng Ma Juanwu Lu Can Cui Sicheng Zhao Xu Cao Wenqian Ye Ziran Wang 24 11 0 25 Oct 2023
Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalization Zhuo Huang Muyang Li Li Shen Jun-chen Yu Chen Gong Bo Han Tongliang Liu OOD 46 8 0 25 Oct 2023
Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks Jules Berman Benjamin Peherstorfer 26 13 0 07 Oct 2023
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen Shengju Qian Haotian Tang Xin Lai Zhijian Liu Song Han Jiaya Jia 42 152 0 21 Sep 2023
Scaled Prompt-Tuning for Few-Shot Natural Language Generation Ting Hu Christoph Meinel Haojin Yang 11 0 0 13 Sep 2023
Ensemble Mask Networks Jonny Luntzel 16 0 0 12 Sep 2023
DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning Zhengxiang Shi Aldo Lipani VLM 28 30 0 11 Sep 2023
Domain Adaptation for Satellite-Borne Hyperspectral Cloud Detection Andrew Du Anh-Dzung Doan Yee Wei Law Tat-Jun Chin 27 2 0 05 Sep 2023
Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models Qiong Wu Wei Yu Yiyi Zhou Shubin Huang Xiaoshuai Sun Rongrong Ji VLM 26 7 0 04 Sep 2023
UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory Haiwen Diao Bo Wan Yuhang Zhang Xuecong Jia Huchuan Lu Long Chen VLM 31 18 0 28 Aug 2023
IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning Feiyu F. Zhang Liangzhi Li Jun-Cheng Chen Zhouqian Jiang Bowen Wang Yiming Qian 51 32 0 23 Aug 2023
SimDA: Simple Diffusion Adapter for Efficient Video Generation Zhen Xing Qi Dai Hang-Rui Hu Zuxuan Wu Yu-Gang Jiang VGen DiffM 35 81 0 18 Aug 2023
Scaling In-Context Demonstrations with Structured Attention Tianle Cai Kaixuan Huang Jason D. Lee Mengdi Wang LRM 31 8 0 05 Jul 2023
Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer Peng Mi Li Shen Tianhe Ren Yiyi Zhou Tianshuo Xu Xiaoshuai Sun Tongliang Liu Rongrong Ji Dacheng Tao AAML 33 2 0 30 Jun 2023
Approximated Prompt Tuning for Vision-Language Pre-trained Models Qiong Wu Shubin Huang Yiyi Zhou Pingyang Dai Annan Shu Guannan Jiang Rongrong Ji VLM VPVLM 25 2 0 27 Jun 2023
Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models Nikhil Kandpal Brian Lester Mohammed Muqeeth Anisha Mascarenhas Monty Evans Vishal Baskaran Tenghao Huang Haokun Liu Colin Raffel VLM 16 10 0 07 Jun 2023