The Impact of LoRA on the Emergence of Clusters in Transformers

23 February 2024

Papers citing "The Impact of LoRA on the Emergence of Clusters in Transformers"

7 / 7 papers shown

Title
The emergence of clusters in self-attention dynamics Borjan Geshkovski Cyril Letrouit Yury Polyanskiy Philippe Rigollet 55 53 0 09 May 2023
Sinkformers: Transformers with Doubly Stochastic Attention Michael E. Sander Pierre Ablin Mathieu Blondel Gabriel Peyré 59 82 0 22 Oct 2021
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth Yihe Dong Jean-Baptiste Cordonnier Andreas Loukas 112 385 0 05 Mar 2021
A Functional Perspective on Learning Symmetric Functions with Neural Networks Aaron Zweig Joan Bruna 35 22 0 16 Aug 2020
The Lipschitz Constant of Self-Attention Hyunjik Kim George Papamakarios A. Mnih 61 145 0 08 Jun 2020
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View Yiping Lu Zhuohan Li Di He Zhiqing Sun Bin Dong Tao Qin Liwei Wang Tie-Yan Liu AI4CE 76 174 0 06 Jun 2019
Neural Ordinary Differential Equations T. Chen Yulia Rubanova J. Bettencourt David Duvenaud AI4CE 406 5,099 0 19 Jun 2018