Title
Small-scale proxies for large-scale Transformer training instabilities Mitchell Wortsman Peter J. Liu Lechao Xiao Katie Everett A. Alemi ... Jascha Narain Sohl-Dickstein Kelvin Xu Jaehoon Lee Justin Gilmer Simon Kornblith 71 98 0 25 Sep 2023
The emergence of clusters in self-attention dynamics Borjan Geshkovski Cyril Letrouit Yury Polyanskiy Philippe Rigollet 53 52 0 09 May 2023
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes Shivam Garg Dimitris Tsipras Percy Liang Gregory Valiant 116 504 0 01 Aug 2022
Streaming Inference for Infinite Non-Stationary Clustering Rylan Schaeffer Gabrielle K. Liu Yilun Du Scott W. Linderman Ila Rani Fiete 50 1 0 02 May 2022
Data Distributional Properties Drive Emergent In-Context Learning in Transformers Stephanie C. Y. Chan Adam Santoro Andrew Kyle Lampinen Jane X. Wang Aaditya K. Singh Pierre Harvey Richemond J. Mcclelland Felix Hill 116 261 0 22 Apr 2022
Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models Beren Millidge Tommaso Salvatori Yuhang Song Thomas Lukasiewicz Rafal Bogacz VLM 42 54 0 09 Feb 2022
Content Addressable Memory Without Catastrophic Forgetting by Heteroassociation with a Fixed Scaffold Sugandha Sharma Sarthak Chandra Ila R. Fiete 94 22 0 01 Feb 2022
Attention Approximates Sparse Distributed Memory Trenton Bricken Cengiz Pehlevan 62 34 0 10 Nov 2021
Unsupervised Learning of Compositional Energy Concepts Yilun Du Shuang Li Yash Sharma J. Tenenbaum Igor Mordatch CoGe OCL 64 80 0 04 Nov 2021
Large Associative Memory Problem in Neurobiology and Machine Learning Dmitry Krotov J. Hopfield 46 136 0 16 Aug 2020
Hopfield Networks is All You Need Hubert Ramsauer Bernhard Schafl Johannes Lehner Philipp Seidl Michael Widrich ... David P. Kreil Michael K Kopp Günter Klambauer Johannes Brandstetter Sepp Hochreiter 83 429 0 16 Jul 2020
On Layer Normalization in the Transformer Architecture Ruibin Xiong Yunchang Yang Di He Kai Zheng Shuxin Zheng Chen Xing Huishuai Zhang Yanyan Lan Liwei Wang Tie-Yan Liu AI4CE 119 988 0 12 Feb 2020
Learning Deep Transformer Models for Machine Translation Qiang Wang Bei Li Tong Xiao Jingbo Zhu Changliang Li Derek F. Wong Lidia S. Chao 70 670 0 05 Jun 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 95 1,894 0 23 Apr 2019
On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models Erik Nijkamp Mitch Hill Tian Han Song-Chun Zhu Ying Nian Wu 51 154 0 29 Mar 2019
Adaptive Input Representations for Neural Language Modeling Alexei Baevski Michael Auli 101 390 0 28 Sep 2018
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 344 10,467 0 21 Jul 2016
Revisiting k-means: New Algorithms via Bayesian Nonparametrics Brian Kulis Michael I. Jordan 71 390 0 02 Nov 2011