Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models

22 May 2023

Guillermo Ortiz-Jiménez

Papers citing "Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models"

40 / 90 papers shown

Title
Knowledge Composition using Task Vectors with Learned Anisotropic Scaling Frederic Z. Zhang Paul Albert Cristian Rodriguez-Opazo Anton van den Hengel Ehsan Abbasnejad MoMe 53 7 0 03 Jul 2024
WARP: On the Benefits of Weight Averaged Rewarded Policies Alexandre Ramé Johan Ferret Nino Vieillard Robert Dadashi Léonard Hussenot Pierre-Louis Cedoz Pier Giuseppe Sessa Sertan Girgin Arthur Douillard Olivier Bachem 56 14 0 24 Jun 2024
MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic Yuyan Zhou Liang Song Bingning Wang Weipeng Chen MoMe 30 15 0 17 Jun 2024
Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging Zhenyi Lu Chenghao Fan Wei Wei Xiaoye Qu Dangyang Chen Yu Cheng MoMe 42 48 0 17 Jun 2024
Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion A. Tang Li Shen Yong Luo Shiwei Liu Han Hu Bo Du MoMe 23 6 0 14 Jun 2024
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation Lu Li T. Zhang Zhiqi Bu Suyuchen Wang Huan He Jie Fu Yonghui Wu Jiang Bian Yong Chen Yoshua Bengio FedML MoMe 97 3 0 11 Jun 2024
FusionBench: A Comprehensive Benchmark of Deep Model Fusion A. Tang Li Shen Yong Luo Han Hu Bo Du Dacheng Tao ELM MoMe VLM 38 21 0 05 Jun 2024
Ensembling Diffusion Models via Adaptive Feature Aggregation Cong Wang Kuan Tian Yonghang Guan Jun Zhang Zhiwei Jiang Fei Shen Xiao Han 42 5 0 27 May 2024
WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models Peng Wang Zexi Li Ningyu Zhang Ziwen Xu Yunzhi Yao Yong-jia Jiang Pengjun Xie Fei Huang Huajun Chen KELM CLL 47 20 0 23 May 2024
Localizing Task Information for Improved Model Merging and Compression Ke Wang Nikolaos Dimitriadis Guillermo Ortiz-Jimenez Franccois Fleuret Pascal Frossard MoMe 33 46 0 13 May 2024
Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning Jing Xu Jingzhao Zhang 36 7 0 04 May 2024
Continuous Language Model Interpolation for Dynamic and Controllable Text Generation Sara Kangaslahti David Alvarez-Melis KELM 29 0 0 10 Apr 2024
A Unified Module for Accelerating STABLE-DIFFUSION: LCM-LORA Ayush Thakur Rashmi Vashisth MoMe 27 2 0 24 Mar 2024
Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization James Oldfield Markos Georgopoulos Grigorios G. Chrysos Christos Tzelepis Yannis Panagakis M. Nicolaou Jiankang Deng Ioannis Patras MoE 42 8 0 19 Feb 2024
On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm Zhanpeng Zhou Zijun Chen Yilan Chen Bo-Wen Zhang Junchi Yan MoMe 19 9 0 06 Feb 2024
Representation Surgery for Multi-Task Model Merging Enneng Yang Li Shen Zhenyi Wang Guibing Guo Xiaojun Chen Xingwei Wang Dacheng Tao MoMe 54 37 0 05 Feb 2024
PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learning Haiyang Guo Fei Zhu Wenzhuo Liu Xu-Yao Zhang Cheng-Lin Liu CLL 27 6 0 04 Jan 2024
A Comprehensive Study of Knowledge Editing for Large Language Models Ningyu Zhang Yunzhi Yao Bo Tian Peng Wang Shumin Deng ... Lei Liang Zhiqiang Zhang Xiao-Jun Zhu Jun Zhou Huajun Chen KELM 41 76 0 02 Jan 2024
Merging by Matching Models in Task Parameter Subspaces Derek Tam Mohit Bansal Colin Raffel MoMe 21 10 0 07 Dec 2023
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization Prateek Yadav Leshem Choshen Colin Raffel Mohit Bansal 32 13 0 22 Nov 2023
LCM-LoRA: A Universal Stable-Diffusion Acceleration Module Simian Luo Yiqin Tan Suraj Patil Daniel Gu Patrick von Platen Apolinário Passos Longbo Huang Jian Li Hang Zhao MoMe 110 144 0 09 Nov 2023
Model Merging by Uncertainty-Based Gradient Matching Nico Daheim Thomas Möllenhoff E. Ponti Iryna Gurevych Mohammad Emtiyaz Khan MoMe FedML 32 43 0 19 Oct 2023
Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective Ming Zhong Chenxin An Weizhu Chen Jiawei Han Pengcheng He 21 9 0 17 Oct 2023
Uncovering hidden geometry in Transformers via disentangling position and context Jiajun Song Yiqiao Zhong 32 10 0 07 Oct 2023
Parameter Efficient Multi-task Model Fusion with Partial Linearization A. Tang Li Shen Yong Luo Yibing Zhan Han Hu Bo Du Yixin Chen Dacheng Tao MoMe 26 30 0 07 Oct 2023
AdaMerging: Adaptive Model Merging for Multi-Task Learning Enneng Yang Zhenyi Wang Li Shen Shiwei Liu Guibing Guo Xingwei Wang Dacheng Tao MoMe 33 95 0 04 Oct 2023
ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale Markus Frohmann Carolin Holtermann Shahed Masoudian Anne Lauscher Navid Rekabsaz 29 2 0 02 Oct 2023
Deep Model Fusion: A Survey Weishi Li Yong Peng Miao Zhang Liang Ding Han Hu Li Shen FedML MoMe 28 51 0 27 Sep 2023
UnIVAL: Unified Model for Image, Video, Audio and Language Tasks Mustafa Shukor Corentin Dancette Alexandre Ramé Matthieu Cord MoMe MLLM 61 42 0 30 Jul 2023
Layer-wise Linear Mode Connectivity Linara Adilova Maksym Andriushchenko Michael Kamp Asja Fischer Martin Jaggi FedML FAtt MoMe 30 15 0 13 Jul 2023
TIES-Merging: Resolving Interference When Merging Models Prateek Yadav Derek Tam Leshem Choshen Colin Raffel Mohit Bansal MoMe 40 250 0 02 Jun 2023
A Kernel-Based View of Language Model Fine-Tuning Sadhika Malladi Alexander Wettig Dingli Yu Danqi Chen Sanjeev Arora VLM 68 60 0 11 Oct 2022
Improving alignment of dialogue agents via targeted human judgements Amelia Glaese Nat McAleese Maja Trkebacz John Aslanides Vlad Firoiu ... John F. J. Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks G. Irving ALM AAML 227 500 0 28 Sep 2022
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 252 313 0 11 Sep 2022
Diverse Weight Averaging for Out-of-Distribution Generalization Alexandre Ramé Matthieu Kirchmeyer Thibaud Rahier A. Rakotomamonjy Patrick Gallinari Matthieu Cord OOD 196 128 0 19 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,915 0 04 Mar 2022
Editing a classifier by rewriting its prediction rules Shibani Santurkar Dimitris Tsipras Mahalaxmi Elango David Bau Antonio Torralba A. Madry KELM 175 89 0 02 Dec 2021
A linearized framework and a new benchmark for model selection for fine-tuning Aditya Deshpande Alessandro Achille Avinash Ravichandran Hao Li L. Zancato Charless C. Fowlkes Rahul Bhotika Stefano Soatto Pietro Perona ALM 112 46 0 29 Jan 2021
Geometric compression of invariant manifolds in neural nets J. Paccolat Leonardo Petrini Mario Geiger Kevin Tyloo M. Wyart MLT 52 34 0 22 Jul 2020
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks Blake Bordelon Abdulkadir Canatar C. Pehlevan 139 201 0 07 Feb 2020