Measuring the Intrinsic Dimension of Objective Landscapes

24 April 2018

Papers citing "Measuring the Intrinsic Dimension of Objective Landscapes"

50 / 106 papers shown

Title
Memory-Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation Fei Wu Jia Hu Geyong Min Shiqiang Wang 22 0 0 16 May 2025
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning Anh Tong Thanh Nguyen-Tang Dongeun Lee Duc Nguyen Toan M. Tran David Hall Cheongwoong Kang Jaesik Choi 35 0 0 03 Mar 2025
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models Yibin Wang Haizhou Shi Ligong Han Dimitris N. Metaxas Hao Wang BDL UQLM 116 8 0 28 Jan 2025
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics Thomas Robert M. Safaryan Ionut-Vlad Modoranu Dan Alistarh ODL 36 2 0 21 Oct 2024
Collaborative and Efficient Personalization with Mixtures of Adaptors Abdulla Jasem Almansoori Samuel Horváth Martin Takáč FedML 46 2 0 04 Oct 2024
Geometric Signatures of Compositionality Across a Language Model's Lifetime Jin Hwa Lee Thomas Jiralerspong Lei Yu Yoshua Bengio Emily Cheng CoGe 87 1 0 02 Oct 2024
Selective Aggregation for Low-Rank Adaptation in Federated Learning Pengxin Guo Shuang Zeng Y. Wang Huijie Fan Feifei Wang Liangqiong Qu FedML 50 11 0 02 Oct 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 75 17 0 25 Jul 2024
SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning Bac Nguyen Stefan Uhlich Fabien Cardinaux Lukas Mauch Marzieh Edraki Aaron Courville OODD CLL VLM 57 3 0 03 Jul 2024
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead Rickard Brüel-Gabrielsson Jiacheng Zhu Onkar Bhardwaj Leshem Choshen Kristjan Greenewald Mikhail Yurochkin Justin Solomon 51 5 0 17 Jun 2024
ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts Samar Khanna Medhanie Irgau David B. Lobell Stefano Ermon VLM 32 4 0 16 Jun 2024
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang Zeguan Xiao Shuo Wang Guanhua Chen Guanhua Chen 44 19 0 13 Jun 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou Shuaiwen Leon Song Jianlong Wu Liqiang Nie Guohao Li 48 6 0 07 Jun 2024
Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation Can Yaras Peng Wang Laura Balzano Qing Qu AI4CE 37 14 0 06 Jun 2024
An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updates Albin Soutif--Cormerais Simone Magistri Joost van de Weijer Andew D. Bagdanov 40 1 0 28 May 2024
Does SGD really happen in tiny subspaces? Minhak Song Kwangjun Ahn Chulhee Yun 73 5 1 25 May 2024
Emergence of a High-Dimensional Abstraction Phase in Language Transformers Emily Cheng Diego Doimo Corentin Kervadec Iuri Macocco Jade Yu Alessandro Laio Marco Baroni 112 11 0 24 May 2024
Machine Unlearning via Null Space Calibration Huiqiang Chen Tianqing Zhu Xin Yu Wanlei Zhou 43 6 0 21 Apr 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 64 76 0 03 Apr 2024
Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottlenecks Tingyu Qu Tinne Tuytelaars Marie-Francine Moens MoE 46 2 0 14 Mar 2024
Uncertainty quantification in fine-tuned LLMs using LoRA ensembles Oleksandr Balabanov Hampus Linander UQCV 41 15 0 19 Feb 2024
Sparse is Enough in Fine-tuning Pre-trained Large Language Models Weixi Song Z. Li Lefei Zhang Hai Zhao Bo Du VLM 29 7 0 19 Dec 2023
Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes Zhen Qin Daoyuan Chen Bingchen Qian Bolin Ding Yaliang Li Shuiguang Deng FedML 45 32 0 11 Dec 2023
PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advance Haichao Sha Ruixuan Liu Yi-xiao Liu Hong Chen 52 1 0 06 Dec 2023
A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA Damjan Kalajdzievski ALM 33 80 0 28 Nov 2023
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models Anton Razzhigaev Matvey Mikhalchuk Elizaveta Goncharova Ivan Oseledets Denis Dimitrov Andrey Kuznetsov 35 7 0 10 Nov 2023
Efficient Cross-Task Prompt Tuning for Few-Shot Conversational Emotion Recognition Yige Xu Zhiwei Zeng Zhiqi Shen VLM 35 3 0 23 Oct 2023
Bridging Information-Theoretic and Geometric Compression in Language Models Emily Cheng Corentin Kervadec Marco Baroni 36 17 0 20 Oct 2023
Decomposed Prompt Tuning via Low-Rank Reparameterization Yao Xiao Lu Xu Jiaxi Li Wei Lu Xiaoli Li VLM 25 6 0 16 Oct 2023
NOLA: Compressing LoRA using Linear Combination of Random Basis Soroush Abbasi Koohpayegani K. Navaneet Parsa Nooralinejad Soheil Kolouri Hamed Pirsiavash 40 12 0 04 Oct 2023
AdaptNet: Policy Adaptation for Physics-Based Character Control Pei Xu Kaixiang Xie Sheldon Andrews P. Kry Michael Neff Morgan McGuire Ioannis Karamouzas Victor Zordan TTA 44 17 0 30 Sep 2023
Prompting or Fine-tuning? A Comparative Study of Large Language Models for Taxonomy Construction Boqi Chen Fandi Yi Dániel Varró 37 16 0 04 Sep 2023
Quantifying lottery tickets under label noise: accuracy, calibration, and complexity V. Arora Daniele Irto Sebastian Goldt G. Sanguinetti 44 2 0 21 Jun 2023
Data Representations' Study of Latent Image Manifolds Ilya Kaufman Omri Azencot 14 7 0 31 May 2023
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning Umang Gupta Aram Galstyan Greg Ver Steeg 11 2 0 30 May 2023
Fine-Tuning Language Models with Just Forward Passes Sadhika Malladi Tianyu Gao Eshaan Nichani Alexandru Damian Jason D. Lee Danqi Chen Sanjeev Arora 43 180 0 27 May 2023
Probing the Purview of Neural Networks via Gradient Analysis Jinsol Lee Charles Lehman Mohit Prabhushankar Ghassan AlRegib 34 7 0 06 Apr 2023
Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation Simone Ciceri Lorenzo Cassani Matteo Osella P. Rotondo P. Pizzochero M. Gherardi 41 7 0 09 Mar 2023
Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning Zhen Wang Yikang Shen Leonid Karlinsky Rogerio Feris Huan Sun Yoon Kim VLM VPVLM 44 108 0 06 Mar 2023
Modular Deep Learning Jonas Pfeiffer Sebastian Ruder Ivan Vulić Edoardo Ponti MoMe OOD 34 73 0 22 Feb 2023
A survey of deep learning optimizers -- first and second order methods Rohan Kashyap ODL 37 6 0 28 Nov 2022
PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization Sanae Lotfi Marc Finzi Sanyam Kapoor Andres Potapczynski Micah Goldblum A. Wilson BDL MLT AI4CE 29 54 0 24 Nov 2022
NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension Xin He Jiangchao Yao Yuxin Wang Zhenheng Tang Ka Chu Cheung Simon See Bo Han Xiaowen Chu 21 9 0 23 Nov 2022
Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States Ziqiao Wang Yongyi Mao 30 10 0 19 Nov 2022
Parameter-Efficient Tuning Makes a Good Classification Head Zhuoyi Yang Ming Ding Yanhui Guo Qingsong Lv Jie Tang VLM 63 14 0 30 Oct 2022
Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuning Jing Yi Weize Chen Yujia Qin Yankai Lin Ning Ding Xu Han Zhiyuan Liu Maosong Sun Jie Zhou 20 2 0 24 Oct 2022
Tunable Complexity Benchmarks for Evaluating Physics-Informed Neural Networks on Coupled Ordinary Differential Equations Alexander New B. Eng A. Timm A. Gearhart 20 4 0 14 Oct 2022
Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask? Mansheej Paul F. Chen Brett W. Larsen Jonathan Frankle Surya Ganguli Gintare Karolina Dziugaite UQCV 42 38 0 06 Oct 2022
Understanding Collapse in Non-Contrastive Siamese Representation Learning Alexander C. Li Alexei A. Efros Deepak Pathak SSL 55 33 0 29 Sep 2022
Extremely Simple Activation Shaping for Out-of-Distribution Detection Andrija Djurisic Nebojsa Bozanic Arjun Ashok Rosanne Liu OODD 172 151 0 20 Sep 2022