Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity

17 October 2018

Papers citing "Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity"

33 / 33 papers shown

Title
On the Complexity of Neural Computation in Superposition Micah Adler Nir Shavit 115 3 0 05 Sep 2024
Analytical Solution of a Three-layer Network with a Matrix Exponential Activation Function Kuo Gai Shihua Zhang FAtt 43 0 0 02 Jul 2024
$\emph{Lifted} RDT based capacity analysis of the 1-hidden layer treelike \emph{sign} perceptrons neural networks$ \emph{Lifted} RDT based capacity analysis of the 1-hidden layer treelike \emph{sign} perceptrons neural networks M. Stojnic 24 1 0 13 Dec 2023
Capacity of the treelike sign perceptrons neural networks with one hidden layer -- RDT based upper bounds M. Stojnic 21 4 0 13 Dec 2023
Minimum width for universal approximation using ReLU networks on compact domain Namjun Kim Chanho Min Sejun Park VLM 29 10 0 19 Sep 2023
Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? T. Kajitsuka Issei Sato 31 16 0 26 Jul 2023
Memorization Capacity of Multi-Head Attention in Transformers Sadegh Mahdavi Renjie Liao Christos Thrampoulidis 26 22 0 03 Jun 2023
Memorization Capacity of Neural Networks with Conditional Computation Erdem Koyuncu 38 4 0 20 Mar 2023
Graph Positional Encoding via Random Feature Propagation Moshe Eliasof Fabrizio Frasca Beatrice Bevilacqua Eran Treister Gal Chechik Haggai Maron 32 18 0 06 Mar 2023
Task Discovery: Finding the Tasks that Neural Networks Generalize on Andrei Atanov Andrei Filatov Teresa Yeo Ajay Sohmshetty Amir Zamir OOD 45 10 0 01 Dec 2022
LU decomposition and Toeplitz decomposition of a neural network Yucong Liu Simiao Jiao Lek-Heng Lim 30 7 0 25 Nov 2022
When Expressivity Meets Trainability: Fewer than $n$ Neurons Can Work Jiawei Zhang Yushun Zhang Mingyi Hong Ruoyu Sun Zhi-Quan Luo 26 10 0 21 Oct 2022
Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Power Binghui Li Jikai Jin Han Zhong J. Hopcroft Liwei Wang OOD 82 27 0 27 May 2022
Randomly Initialized One-Layer Neural Networks Make Data Linearly Separable Promit Ghosal Srinath Mahankali Yihang Sun MLT 24 4 0 24 May 2022
$Training Fully Connected Neural Networks is $\exists\mathbb{R}$-Complete$ Training Fully Connected Neural Networks is $\exists\mathbb{R}$ -Complete Daniel Bertschinger Christoph Hertrich Paul Jungeblut Tillmann Miltzow Simon Weber OffRL 59 30 0 04 Apr 2022
Selective Network Linearization for Efficient Private Inference Minsu Cho Ameya Joshi S. Garg Brandon Reagen C. Hegde 6 43 0 04 Feb 2022
Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer Beatrice Acciaio Anastasis Kratsios G. Pammer OOD 52 20 0 31 Jan 2022
Improved Overparametrization Bounds for Global Convergence of Stochastic Gradient Descent for Shallow Neural Networks Bartlomiej Polaczyk J. Cyranka ODL 33 3 0 28 Jan 2022
Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views? M. Farrell Blake Bordelon Shubhendu Trivedi C. Pehlevan 18 5 0 14 Oct 2021
Equivariant Subgraph Aggregation Networks Beatrice Bevilacqua Fabrizio Frasca Derek Lim Balasubramaniam Srinivasan Chen Cai G. Balamurugan M. Bronstein Haggai Maron 53 175 0 06 Oct 2021
A Universal Law of Robustness via Isoperimetry Sébastien Bubeck Mark Sellke 13 213 0 26 May 2021
A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks Asaf Noy Yi Tian Xu Y. Aflalo Lihi Zelnik-Manor R. L. Jin 33 3 0 12 Jan 2021
Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks Quynh N. Nguyen Marco Mondelli Guido Montúfar 25 81 0 21 Dec 2020
From Local Structures to Size Generalization in Graph Neural Networks Gilad Yehudai Ethan Fetaya E. Meirom Gal Chechik Haggai Maron GNN AI4CE 169 123 0 17 Oct 2020
A law of robustness for two-layers neural networks Sébastien Bubeck Yuanzhi Li Dheeraj M. Nagaraj 27 57 0 30 Sep 2020
The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training Andrea Montanari Yiqiao Zhong 47 95 0 25 Jul 2020
Minimum Width for Universal Approximation Sejun Park Chulhee Yun Jaeho Lee Jinwoo Shin 33 121 0 16 Jun 2020
Approximation in shift-invariant spaces with deep ReLU neural networks Yunfei Yang Zhen Li Yang Wang 34 14 0 25 May 2020
Memory capacity of neural networks with threshold and ReLU activations Roman Vershynin 31 21 0 20 Jan 2020
Universal Approximation with Deep Narrow Networks Patrick Kidger Terry Lyons 31 324 0 21 May 2019
NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representation Anastasis Kratsios Cody B. Hyndman OOD 25 17 0 31 Aug 2018
Global optimality conditions for deep neural networks Chulhee Yun S. Sra Ali Jadbabaie 128 117 0 08 Jul 2017
Benefits of depth in neural networks Matus Telgarsky 148 602 0 14 Feb 2016