Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning

18 December 2023

Jieru Mei

Papers citing "Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning"

32 / 32 papers shown

Title
Concept Drift Guided LayerNorm Tuning for Efficient Multimodal Metaphor Identification Wenhao Qian Zhenzhen Hu Zijie Song Jia Li 14 0 0 16 May 2025
OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning Xianhang Li Y. Liu Haoqin Tu Hongru Zhu Cihang Xie VLM 180 0 0 07 May 2025
VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection Hao Cheng Zhiwei Zhao Yichao He Zhenzhen Hu Jia Li Hao Wu Richang Hong 46 0 0 05 May 2025
Segment Any Crack: Deep Semantic Segmentation Adaptation for Crack Detection Ghodsiyeh Rostami Po-Han Chen Mahdi S. Hosseini 29 0 0 19 Apr 2025
Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities Raman Dutt Harleen Hanspal Guoxuan Xia Petru-Daniel Tudosiu Alexander Black Yongxin Yang Jingyu Sun Sarah Parisot MoE 43 0 0 28 Mar 2025
DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation Massimo Bini Leander Girrbach Zeynep Akata 45 0 0 23 Mar 2025
Parameter-Efficient Adaptation of Geospatial Foundation Models through Embedding Deflection Romain Thoreau Valerio Marsocci Dawa Derksen AI4CE 77 2 0 12 Mar 2025
Learning to Reason from Feedback at Test-Time Yanyang Li M. Lyu Liwei Wang LRM 36 1 0 16 Feb 2025
Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models Christopher Simic Korbinian Riedhammer Tobias Bocklet 93 0 0 03 Feb 2025
Parameter-Efficient Fine-Tuning for Foundation Models Dan Zhang Tao Feng Lilong Xue Yuandong Wang Yuxiao Dong J. Tang 46 8 0 23 Jan 2025
A Hessian-informed hyperparameter optimization for differential learning rate Shiyun Xu Zhiqi Bu Yiliang Zhang Ian Barnett 39 1 0 12 Jan 2025
A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine Hanguang Xiao Feizhong Zhou X. Liu Tianqi Liu Zhipeng Li Xin Liu Xiaoxuan Huang AILaw LM&MA LRM 61 18 0 31 Dec 2024
PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning Shenghui Li Edith C.H. Ngai Fanghua Ye Thiemo Voigt SILM 90 6 0 28 Nov 2024
ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning Zhiwei Hao Jianyuan Guo Li Shen Yong Luo Han Hu Yonggang Wen VLM 26 0 0 23 Oct 2024
AERO: Softmax-Only LLMs for Efficient Private Inference N. Jha Brandon Reagen 32 1 0 16 Oct 2024
Revisited Large Language Model for Time Series Analysis through Modality Alignment L. Zheng Chang Dong W. Zhang Lin Yue Miao Xu Olaf Maennel Weitong Chen AI4TS 172 2 0 16 Oct 2024
RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates Md. Kowsher Tara Esmaeilbeig Chun-Nam Yu Mojtaba Soltanalian Niloofar Yousefi 32 0 0 14 Oct 2024
ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models N. Jha Brandon Reagen OffRL AI4CE 33 0 0 12 Oct 2024
PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization Yao Ni Shan Zhang Piotr Koniusz 181 2 0 25 Sep 2024
Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models Divij Gupta Anubhav Bhatti Surajsinh Parmar AI4TS 34 1 0 17 Sep 2024
HRP: Human Affordances for Robotic Pre-Training Mohan Kumar Srirama Sudeep Dasari Shikhar Bahl Abhinav Gupta 41 15 0 26 Jul 2024
ASPS: Augmented Segment Anything Model for Polyp Segmentation Huiqian Li Dingwen Zhang Jieru Yao Longfei Han Zhongyu Li J. Han 53 10 0 30 Jun 2024
Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning Bingchen Zhao Yongshuo Zong Letian Zhang Timothy Hospedales VLM 33 15 0 18 Jun 2024
Effectiveness of Vision Language Models for Open-world Single Image Test Time Adaptation Manogna Sreenivas Soma Biswas VLM 48 0 0 01 Jun 2024
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization Zhe Li Bicheng Ying Zidong Liu Haibo Yang Haibo Yang FedML 59 3 0 24 May 2024
Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models Jiawei Chen Dingkang Yang Yue Jiang Mingcheng Li Jinjie Wei Xiaolu Hou Lihua Zhang 56 6 0 25 Apr 2024
Lowering PyTorch's Memory Consumption for Selective Differentiation Samarth Bhatia Felix Dangel 24 1 0 15 Apr 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 318 0 21 Mar 2024
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 208 905 0 27 Apr 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 302 4,261 0 30 Jan 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 351 12,003 0 04 Mar 2022
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts Soravit Changpinyo P. Sharma Nan Ding Radu Soricut VLM 299 1,084 0 17 Feb 2021