Fine-tuning can cripple your foundation model; preserving features may be the solution

25 August 2023

Papers citing "Fine-tuning can cripple your foundation model; preserving features may be the solution"

30 / 30 papers shown

Title
MoCLIP: Motion-Aware Fine-Tuning and Distillation of CLIP for Human Motion Generation Gabriel Maldonado Armin Danesh Pazho Ghazal Alinezhad Noghre Vinit Katariya Hamed Tabkhi CLIP VGen 20 0 0 16 May 2025
Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety Zihan Guan Mengxuan Hu Ronghang Zhu Sheng Li Anil Vullikanti AAML 31 0 0 11 May 2025
Vanishing Depth: A Depth Adapter with Positional Depth Encoding for Generalized Image Encoders Paul Koch Jörg Krüger Ankit Chowdhury O. Heimann MDE 55 0 0 25 Mar 2025
SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging Aladin Djuhera S. Kadhe Farhan Ahmed Syed Zawad Holger Boche MoMe 51 0 0 21 Mar 2025
Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation Yishuo Wang Tiansheng Huang Li Shen H. Yao Haotian Luo Rui Liu Naiqiang Tan Jiaxing Huang Dacheng Tao AAML MoMe CLL 111 2 0 30 Jan 2025
Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates Kaifeng Lyu Haoyu Zhao Xinran Gu Dingli Yu Anirudh Goyal Sanjeev Arora ALM 82 44 0 20 Jan 2025
PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning Shenghui Li Edith C. H. Ngai Fanghua Ye Thiemo Voigt SILM 90 6 0 28 Nov 2024
FEET: A Framework for Evaluating Embedding Techniques Simon A. Lee John Lee Jeffrey N. Chiang 34 3 0 02 Nov 2024
Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model Divyanshu Aggarwal Sankarshan Damle Navin Goyal Satya Lokam Sunayana Sitaram CLL 23 0 0 21 Oct 2024
Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation Guozhi Liu Weiwei Lin Tiansheng Huang Ruichao Mo Qi Mu Li Shen AAML 63 10 0 13 Oct 2024
SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection Han Shen Pin-Yu Chen Payel Das Tianyi Chen ALM 26 11 0 09 Oct 2024
Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey Tiansheng Huang Sihao Hu Fatih Ilhan Selim Furkan Tekin Ling Liu AAML 42 21 0 26 Sep 2024
Minimizing Embedding Distortion for Robust Out-of-Distribution Performance Tom Shaked Yuval Goldman Oran Shayer OODD 25 0 0 11 Sep 2024
Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Tiansheng Huang Gautam Bhattacharya Pratik Joshi Josh Kimball Ling Liu AAML MoMe 49 16 0 18 Aug 2024
ICLGuard: Controlling In-Context Learning Behavior for Applicability Authorization Wai Man Si Michael Backes Yang Zhang 43 1 0 09 Jul 2024
Safety Alignment Should Be Made More Than Just a Few Tokens Deep Xiangyu Qi Ashwinee Panda Kaifeng Lyu Xiao Ma Subhrajit Roy Ahmad Beirami Prateek Mittal Peter Henderson 47 73 0 10 Jun 2024
kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies Zhongrui Gui Shuyang Sun Runjia Li Jianhao Yuan Zhaochong An Karsten Roth Ameya Prabhu Philip H. S. Torr VLM CLL 32 6 0 15 Apr 2024
Vaccine: Perturbation-aware Alignment for Large Language Model Tiansheng Huang Sihao Hu Ling Liu 50 33 0 02 Feb 2024
AutoFT: Learning an Objective for Robust Fine-Tuning Caroline Choi Yoonho Lee Annie S. Chen Allan Zhou Aditi Raghunathan Chelsea Finn OOD 50 0 0 18 Jan 2024
When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations Aleksandar Petrov Philip H. S. Torr Adel Bibi VPVLM 30 21 0 30 Oct 2023
FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning Kun Song Huimin Ma Bochao Zou Huishuai Zhang Weiran Huang 18 10 0 23 Oct 2023
Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models Zangwei Zheng Mingyu Ma Kai Wang Ziheng Qin Xiangyu Yue Yang You CLL VLM 102 68 0 12 Mar 2023
CLIP model is an Efficient Continual Learner Vishal G. Thengane Salman Khan Munawar Hayat F. Khan BDL VLM CLL 112 46 0 06 Oct 2022
Self-Distillation for Further Pre-training of Transformers Seanie Lee Minki Kang Juho Lee Sung Ju Hwang Kenji Kawaguchi 45 8 0 30 Sep 2022
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models Manli Shu Weili Nie De-An Huang Zhiding Yu Tom Goldstein Anima Anandkumar Chaowei Xiao VLM VPVLM 186 282 0 15 Sep 2022
Fine-tuned Language Models are Continual Learners Thomas Scialom Tuhin Chakrabarty Smaranda Muresan CLL LRM 145 117 0 24 May 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 305 7,443 0 11 Nov 2021
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling Renrui Zhang Rongyao Fang Wei Zhang Peng Gao Kunchang Li Jifeng Dai Yu Qiao Hongsheng Li VLM 192 385 0 06 Nov 2021
Learning to Prompt for Vision-Language Models Kaiyang Zhou Jingkang Yang Chen Change Loy Ziwei Liu VPVLM CLIP VLM 348 2,271 0 02 Sep 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 310 3,708 0 11 Feb 2021