v1v2 (latest)

How Smooth Is Attention?

22 December 2023

Pierre Ablin

Papers citing "How Smooth Is Attention?"

29 / 29 papers shown

Title
A mathematical perspective on Transformers Borjan Geshkovski Cyril Letrouit Yury Polyanskiy Philippe Rigollet EDL AI4CE 98 46 0 17 Dec 2023
The emergence of clusters in self-attention dynamics Borjan Geshkovski Cyril Letrouit Yury Polyanskiy Philippe Rigollet 77 56 0 09 May 2023
Sinkformers: Transformers with Doubly Stochastic Attention Michael E. Sander Pierre Ablin Mathieu Blondel Gabriel Peyré 76 82 0 22 Oct 2021
On the Expressive Power of Self-Attention Matrices Valerii Likhosherstov K. Choromanski Adrian Weller 84 36 0 07 Jun 2021
Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networks George Dasoulas Kevin Scaman Aladin Virmaux GNN 63 39 0 08 Mar 2021
Globally-Robust Neural Networks Klas Leino Zifan Wang Matt Fredrikson AAML OOD 131 130 0 16 Feb 2021
On the Regularity of Attention James Vuckovic A. Baratin Rémi Tachet des Combes 34 7 0 10 Feb 2021
A case for new neural network smoothness constraints Mihaela Rosca T. Weber Arthur Gretton S. Mohamed AAML 98 50 0 14 Dec 2020
Point Transformer Nico Engel Vasileios Belagiannis Klaus C. J. Dietmayer 3DPC 181 2,003 0 02 Nov 2020
A Functional Perspective on Learning Symmetric Functions with Neural Networks Aaron Zweig Joan Bruna 46 22 0 16 Aug 2020
The Lipschitz Constant of Self-Attention Hyunjik Kim George Papamakarios A. Mnih 77 146 0 08 Jun 2020
On Layer Normalization in the Transformer Architecture Ruibin Xiong Yunchang Yang Di He Kai Zheng Shuxin Zheng Chen Xing Huishuai Zhang Yanyan Lan Liwei Wang Tie-Yan Liu AI4CE 142 995 0 12 Feb 2020
Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks Mahyar Fazlyab Alexander Robey Hamed Hassani M. Morari George J. Pappas 96 460 0 12 Jun 2019
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View Yiping Lu Zhuohan Li Di He Zhiqing Sun Bin Dong Tao Qin Liwei Wang Tie-Yan Liu AI4CE 78 174 0 06 Jun 2019
Residual Flows for Invertible Generative Modeling Ricky T. Q. Chen Jens Behrmann David Duvenaud J. Jacobsen BDL TPM DRL 111 377 0 06 Jun 2019
Stochastic Deep Networks Gwendoline de Bie Gabriel Peyré Marco Cuturi 81 21 0 19 Nov 2018
Sorting out Lipschitz function approximation Cem Anil James Lucas Roger C. Grosse 86 324 0 13 Nov 2018
Neural Ordinary Differential Equations T. Chen Yulia Rubanova J. Bettencourt David Duvenaud AI4CE 417 5,156 0 19 Jun 2018
Lipschitz regularity of deep neural networks: analysis and efficient estimation Kevin Scaman Aladin Virmaux 83 529 0 28 May 2018
Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks Yusuke Tsuzuku Issei Sato Masashi Sugiyama AAML 105 308 0 12 Feb 2018
Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach Tsui-Wei Weng Huan Zhang Pin-Yu Chen Jinfeng Yi D. Su Yupeng Gao Cho-Jui Hsieh Luca Daniel AAML 83 468 0 31 Jan 2018
Spectrally-normalized margin bounds for neural networks Peter L. Bartlett Dylan J. Foster Matus Telgarsky ODL 207 1,224 0 26 Jun 2017
Parseval Networks: Improving Robustness to Adversarial Examples Moustapha Cissé Piotr Bojanowski Edouard Grave Yann N. Dauphin Nicolas Usunier AAML 138 808 0 28 Apr 2017
Adversarial Machine Learning at Scale Alexey Kurakin Ian Goodfellow Samy Bengio AAML 472 3,147 0 04 Nov 2016
Towards Evaluating the Robustness of Neural Networks Nicholas Carlini D. Wagner OOD AAML 266 8,579 0 16 Aug 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 416 10,526 0 21 Jul 2016
Explaining and Harnessing Adversarial Examples Ian Goodfellow Jonathon Shlens Christian Szegedy AAML GAN 280 19,107 0 20 Dec 2014
Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio AIMat 575 27,325 0 01 Sep 2014
Intriguing properties of neural networks Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna D. Erhan Ian Goodfellow Rob Fergus AAML 280 14,961 1 21 Dec 2013