Unraveling Attention via Convex Duality: Analysis and Interpretations of
Vision Transformers

Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers

17 May 2022

Batu Mehmet Ozturkler

Morteza Mardani

Papers citing "Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers"

10 / 10 papers shown

Title
Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers Lorenzo Tiberi Francesca Mignacco Kazuki Irie H. Sompolinsky 44 6 0 24 May 2024
Implicit Bias and Fast Convergence Rates for Self-attention Bhavya Vasudeva Puneesh Deora Christos Thrampoulidis 34 13 0 08 Feb 2024
Memorization Capacity of Multi-Head Attention in Transformers Sadegh Mahdavi Renjie Liao Christos Thrampoulidis 26 22 0 03 Jun 2023
Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions Aaron Mishkin Arda Sahiner Mert Pilanci OffRL 77 30 0 02 Feb 2022
Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks Tolga Ergen Mert Pilanci 32 16 0 18 Oct 2021
Parallel Deep Neural Networks Have Zero Duality Gap Yifei Wang Tolga Ergen Mert Pilanci 79 10 0 13 Oct 2021
Is Attention Better Than Matrix Decomposition? Zhengyang Geng Meng-Hao Guo Hongxu Chen Xia Li Ke Wei Zhouchen Lin 62 137 0 09 Sep 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 271 2,603 0 04 May 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 326 5,785 0 29 Apr 2021
Fourier Neural Operator for Parametric Partial Differential Equations Zong-Yi Li Nikola B. Kovachki Kamyar Azizzadenesheli Burigede Liu K. Bhattacharya Andrew M. Stuart Anima Anandkumar AI4CE 223 2,287 0 18 Oct 2020