Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

22 December 2020

Luke Zettlemoyer

Papers citing "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning"

50 / 60 papers shown

Title
Accelerating Learned Image Compression Through Modeling Neural Training Dynamics Yichi Zhang Zhihao Duan Yuning Huang Fengqing Zhu 106 0 0 23 May 2025
ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation Mohammad Kasra Habib Daniel Graziotin Stefan Wagner 85 0 0 23 May 2025
TRACE for Tracking the Emergence of Semantic Representations in Transformers Nura Aljaafari Danilo S. Carvalho André Freitas 45 0 0 23 May 2025
ZenFlow: Enabling Stall-Free Offloading Training via Asynchronous Updates Tingfeng Lan Yusen Wu Bin Ma Zhaoyuan Su Rui Yang Tekin Bicer Dong Li Yue Cheng 106 0 0 18 May 2025
FISH-Tuning: Enhancing PEFT Methods with Fisher Information Kang Xue Ming Dong Xinhui Tu Tingting He 97 0 0 05 Apr 2025
Quantum-PEFT: Ultra parameter-efficient fine-tuning Toshiaki Koike-Akino F. Tonin Yongtao Wu Frank Zhengqing Wu Leyla Naz Candogan Volkan Cevher MQ 106 5 0 07 Mar 2025
Liger: Linearizing Large Language Models to Gated Recurrent Structures Disen Lan Weigao Sun Jiaxi Hu Jusen Du Yu Cheng 85 0 0 03 Mar 2025
PaCA: Partial Connection Adaptation for Efficient Fine-Tuning Sunghyeon Woo Sol Namkung Sunwoo Lee Inho Jeong Beomseok Kim Dongsuk Jeon 70 0 0 28 Feb 2025
Low Tensor-Rank Adaptation of Kolmogorov--Arnold Networks Yihang Gao Michael K. Ng Vincent Y. F. Tan 145 0 0 17 Feb 2025
Explaining Context Length Scaling and Bounds for Language Models Jingzhe Shi Qinwei Ma Hongyi Liu Hang Zhao Jeng-Neng Hwang Lei Li LRM 156 3 0 03 Feb 2025
Memory-Efficient Fine-Tuning of Transformers via Token Selection Antoine Simoulin Namyong Park Xiaoyi Liu Grey Yang 142 0 0 31 Jan 2025
DAGPrompT: Pushing the Limits of Graph Prompting with a Distribution-aware Graph Prompt Tuning Approach Qin Chen Liang Wang Bo Zheng Guojie Song 81 3 0 28 Jan 2025
MADation: Face Morphing Attack Detection with Foundation Models Eduarda Caldeira Guray Ozgur Tahar Chettaoui Marija Ivanovska Peter Peer Fadi Boutros Vitomir Štruc Naser Damer CVBM 56 2 1 28 Jan 2025
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models Yibin Wang Haizhou Shi Ligong Han Dimitris N. Metaxas Hao Wang BDL UQLM 127 8 0 28 Jan 2025
ABXI: Invariant Interest Adaptation for Task-Guided Cross-Domain Sequential Recommendation Qingtian Bian Marcus Vinícius de Carvalho Tieying Li Jiaxing Xu Hui Fang Yiping Ke 66 0 0 25 Jan 2025
Parameter-Efficient Fine-Tuning for Foundation Models Dan Zhang Tao Feng Lilong Xue Yuandong Wang Yuxiao Dong J. Tang 107 9 0 23 Jan 2025
Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards Omar Erak Nouf Alabbasi Omar Alhussein Ismail Lotfi Amr Hussein Sami Muhaidat Merouane Debbah RALM 94 5 0 17 Jan 2025
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Akhiad Bercovich Tomer Ronen Talor Abramovich Nir Ailon Nave Assaf ... Ido Shahaf Oren Tropp Omer Ullman Argov Ran Zilberstein Ran El-Yaniv 133 3 0 28 Nov 2024
FRoundation: Are Foundation Models Ready for Face Recognition? Tahar Chettaoui Naser Damer Fadi Boutros CVBM 52 5 0 31 Oct 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies Liwen Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 81 5 0 24 Oct 2024
Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations M. Germán-Morales A. J. Rivera-Rivas M. J. del Jesus Díaz C. J. Carmona AI4TS AI4CE 139 0 0 15 Oct 2024
Collaborative and Efficient Personalization with Mixtures of Adaptors Abdulla Jasem Almansoori Samuel Horváth Martin Takáč FedML 64 3 0 04 Oct 2024
Selective Aggregation for Low-Rank Adaptation in Federated Learning Pengxin Guo Shuang Zeng Y. Wang Huijie Fan Feifei Wang Liangqiong Qu FedML 71 12 0 02 Oct 2024
Geometric Signatures of Compositionality Across a Language Model's Lifetime Jin Hwa Lee Thomas Jiralerspong Lei Yu Yoshua Bengio Emily Cheng CoGe 99 3 0 02 Oct 2024
Attention layers provably solve single-location regression Pierre Marion Raphael Berthier Gérard Biau Claire Boyer 320 4 0 02 Oct 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 86 25 0 25 Jul 2024
Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences Nikolaos Dimitriadis Pascal Frossard François Fleuret MoE 131 8 0 10 Jul 2024
ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts Samar Khanna Medhanie Irgau David B. Lobell Stefano Ermon VLM 68 4 0 16 Jun 2024
Enhancing Domain Adaptation through Prompt Gradient Alignment Hoang Phan Lam C. Tran Quyen Tran Trung Le 64 0 0 13 Jun 2024
Talking Heads: Understanding Inter-layer Communication in Transformer Language Models Jack Merullo Carsten Eickhoff Ellie Pavlick 84 14 0 13 Jun 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou Shuaiwen Leon Song Jianlong Wu Liqiang Nie Guohao Li 54 11 0 07 Jun 2024
An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updates Albin Soutif--Cormerais Simone Magistri Joost van de Weijer Andew D. Bagdanov 51 1 0 28 May 2024
Emergence of a High-Dimensional Abstraction Phase in Language Transformers Emily Cheng Diego Doimo Corentin Kervadec Iuri Macocco Jade Yu Alessandro Laio Marco Baroni 117 13 0 24 May 2024
LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks Michelle Halbheer Dominik J. Mühlematter Alexander Becker Dominik Narnhofer Helge Aasen Konrad Schindler Mehmet Özgür Türkoglu UQCV 60 2 0 23 May 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 91 92 0 03 Apr 2024
Uncertainty quantification in fine-tuned LLMs using LoRA ensembles Oleksandr Balabanov Hampus Linander UQCV 73 16 0 19 Feb 2024
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs Max Zimmer Megi Andoni Christoph Spiegel Sebastian Pokutta VLM 74 10 0 23 Dec 2023
Muppet: Massive Multi-task Representations with Pre-Finetuning Armen Aghajanyan Anchit Gupta Akshat Shrivastava Xilun Chen Luke Zettlemoyer Sonal Gupta 49 267 0 26 Jan 2021
Better Fine-Tuning by Reducing Representational Collapse Armen Aghajanyan Akshat Shrivastava Anchit Gupta Naman Goyal Luke Zettlemoyer S. Gupta AAML 59 209 0 06 Aug 2020
Pre-training via Paraphrasing M. Lewis Marjan Ghazvininejad Gargi Ghosh Armen Aghajanyan Sida I. Wang Luke Zettlemoyer AIMat 72 160 0 26 Jun 2020
When BERT Plays the Lottery, All Tickets Are Winning Sai Prasanna Anna Rogers Anna Rumshisky MILM 37 187 0 01 May 2020
Unsupervised Cross-lingual Representation Learning at Scale Alexis Conneau Kartikay Khandelwal Naman Goyal Vishrav Chaudhary Guillaume Wenzek Francisco Guzmán Edouard Grave Myle Ott Luke Zettlemoyer Veselin Stoyanov 126 6,454 0 05 Nov 2019
Adversarial NLI: A New Benchmark for Natural Language Understanding Yixin Nie Adina Williams Emily Dinan Joey Tianyi Zhou Jason Weston Douwe Kiela 89 991 0 31 Oct 2019
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension M. Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdel-rahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer AIMat VLM 105 10,720 0 29 Oct 2019
Evaluating Lottery Tickets Under Distributional Shifts Shrey Desai Hongyuan Zhan Ahmed Aly UQCV OOD 39 41 0 28 Oct 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 243 19,824 0 23 Oct 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 242 6,420 0 26 Sep 2019
Visualizing and Understanding the Effectiveness of BERT Y. Hao Li Dong Furu Wei Ke Xu 80 183 0 15 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 366 24,160 0 26 Jul 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 158 8,386 0 19 Jun 2019