Memorization in Self-Supervised Learning Improves Downstream Generalization

19 January 2024

Wenhao Wang

Muhammad Ahmad Kaleem

Papers citing "Memorization in Self-Supervised Learning Improves Downstream Generalization"

24 / 24 papers shown

Title
The SSL Interplay: Augmentations, Inductive Bias, and Generalization Vivien A. Cabannes B. Kiani Randall Balestriero Yann LeCun A. Bietti SSL 60 32 0 06 Feb 2023
Measuring Forgetting of Memorized Training Examples Matthew Jagielski Om Thakkar Florian Tramèr Daphne Ippolito Katherine Lee ... Eric Wallace Shuang Song Abhradeep Thakurta Nicolas Papernot Chiyuan Zhang TDI 111 108 0 30 Jun 2022
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models Kushal Tirumala Aram H. Markosyan Luke Zettlemoyer Armen Aghajanyan TDI 99 191 0 22 May 2022
Contrasting the landscape of contrastive and non-contrastive learning Ashwini Pokle Jinjin Tian Yuchen Li Andrej Risteski SSL 70 28 0 29 Mar 2022
Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap Yifei Wang Qi Zhang Yisen Wang Jiansheng Yang Zhouchen Lin 58 101 0 25 Mar 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 427 7,705 0 11 Nov 2021
Understanding Dimensional Collapse in Contrastive Self-supervised Learning Li Jing Pascal Vincent Yann LeCun Yuandong Tian SSL 111 347 0 18 Oct 2021
EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learning Hongbin Liu Jinyuan Jia Wenjie Qu Neil Zhenqiang Gong 32 95 0 25 Aug 2021
On the Memorization Properties of Contrastive Learning Ildus Sadrtdinov Nadezhda Chirkova E. Lobacheva SSL 21 3 0 21 Jul 2021
Deep Learning on a Data Diet: Finding Important Examples Early in Training Mansheej Paul Surya Ganguli Gintare Karolina Dziugaite 105 456 0 15 Jul 2021
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning Adrien Bardes Jean Ponce Yann LeCun SSL DML 149 931 0 11 May 2021
On Feature Decorrelation in Self-Supervised Learning Tianyu Hua Wenxiao Wang Zihui Xue Sucheng Ren Yue Wang Hang Zhao SSL OOD 166 191 0 02 May 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 611 6,029 0 29 Apr 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 434 1,906 0 14 Dec 2020
Exploring Simple Siamese Representation Learning Xinlei Chen Kaiming He SSL 245 4,036 0 20 Nov 2020
For self-supervised learning, Rationality implies generalization, provably Yamini Bansal Gal Kaplun Boaz Barak OOD SSL 93 22 0 16 Oct 2020
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation Vitaly Feldman Chiyuan Zhang TDI 130 459 0 09 Aug 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 360 20,053 0 23 Oct 2019
A Closer Look at Memorization in Deep Networks Devansh Arpit Stanislaw Jastrzebski Nicolas Ballas David M. Krueger Emmanuel Bengio ... Tegan Maharaj Asja Fischer Aaron Courville Yoshua Bengio Simon Lacoste-Julien TDI 118 1,814 0 16 Jun 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 306 4,623 0 10 Nov 2016
Membership Inference Attacks against Machine Learning Models Reza Shokri M. Stronati Congzheng Song Vitaly Shmatikov SLR MIALM MIACV 228 4,103 0 18 Oct 2016
Semantic Understanding of Scenes through the ADE20K Dataset Bolei Zhou Hang Zhao Xavier Puig Tete Xiao Sanja Fidler Adela Barriuso Antonio Torralba SSeg 364 1,865 0 18 Aug 2016
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.3K 39,472 0 01 Sep 2014
Representation Learning: A Review and New Perspectives Yoshua Bengio Aaron Courville Pascal Vincent OOD SSL 216 12,422 0 24 Jun 2012