v1v2v3 (latest)

Memorization in Self-Supervised Learning Improves Downstream Generalization

19 January 2024

Wenhao Wang

Muhammad Ahmad Kaleem

Papers citing "Memorization in Self-Supervised Learning Improves Downstream Generalization"

29 / 29 papers shown

Title
The SSL Interplay: Augmentations, Inductive Bias, and Generalization Vivien A. Cabannes B. Kiani Randall Balestriero Yann LeCun A. Bietti SSL 77 33 0 06 Feb 2023
Measuring Forgetting of Memorized Training Examples Matthew Jagielski Om Thakkar Florian Tramèr Daphne Ippolito Katherine Lee ... Eric Wallace Shuang Song Abhradeep Thakurta Nicolas Papernot Chiyuan Zhang TDI 126 110 0 30 Jun 2022
The Privacy Onion Effect: Memorization is Relative Nicholas Carlini Matthew Jagielski Chiyuan Zhang Nicolas Papernot Andreas Terzis Florian Tramèr PILM MIACV 116 110 0 21 Jun 2022
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models Kushal Tirumala Aram H. Markosyan Luke Zettlemoyer Armen Aghajanyan TDI 110 196 0 22 May 2022
Contrasting the landscape of contrastive and non-contrastive learning Ashwini Pokle Jinjin Tian Yuchen Li Andrej Risteski SSL 83 28 0 29 Mar 2022
Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap Yifei Wang Qi Zhang Yisen Wang Jiansheng Yang Zhouchen Lin 68 102 0 25 Mar 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 470 7,819 0 11 Nov 2021
Understanding Dimensional Collapse in Contrastive Self-supervised Learning Li Jing Pascal Vincent Yann LeCun Yuandong Tian SSL 120 358 0 18 Oct 2021
EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learning Hongbin Liu Jinyuan Jia Wenjie Qu Neil Zhenqiang Gong 39 98 0 25 Aug 2021
On the Memorization Properties of Contrastive Learning Ildus Sadrtdinov Nadezhda Chirkova E. Lobacheva SSL 29 3 0 21 Jul 2021
Deep Learning on a Data Diet: Finding Important Examples Early in Training Mansheej Paul Surya Ganguli Gintare Karolina Dziugaite 117 461 0 15 Jul 2021
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning Adrien Bardes Jean Ponce Yann LeCun SSL DML 153 944 0 11 May 2021
On Feature Decorrelation in Self-Supervised Learning Tianyu Hua Wenxiao Wang Zihui Xue Sucheng Ren Yue Wang Hang Zhao SSL OOD 183 195 0 02 May 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 708 6,127 0 29 Apr 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown Basel Alomair Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 509 1,946 0 14 Dec 2020
When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning? Gavin Brown Mark Bun Vitaly Feldman Adam D. Smith Kunal Talwar 298 100 0 11 Dec 2020
Exploring Simple Siamese Representation Learning Xinlei Chen Kaiming He SSL 258 4,072 0 20 Nov 2020
For self-supervised learning, Rationality implies generalization, provably Yamini Bansal Gal Kaplun Boaz Barak OOD SSL 107 22 0 16 Oct 2020
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation Vitaly Feldman Chiyuan Zhang TDI 202 470 0 09 Aug 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 375 18,859 0 13 Feb 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 462 20,317 0 23 Oct 2019
Does Learning Require Memorization? A Short Tale about a Long Tail Vitaly Feldman TDI 132 498 0 12 Jun 2019
A Closer Look at Memorization in Deep Networks Devansh Arpit Stanislaw Jastrzebski Nicolas Ballas David M. Krueger Emmanuel Bengio ... Tegan Maharaj Asja Fischer Aaron Courville Yoshua Bengio Simon Lacoste-Julien TDI 128 1,825 0 16 Jun 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 345 4,636 0 10 Nov 2016
Membership Inference Attacks against Machine Learning Models Reza Shokri M. Stronati Congzheng Song Vitaly Shmatikov SLR MIALM MIACV 272 4,159 0 18 Oct 2016
Semantic Understanding of Scenes through the ADE20K Dataset Bolei Zhou Hang Zhao Xavier Puig Tete Xiao Sanja Fidler Adela Barriuso Antonio Torralba SSeg 404 1,890 0 18 Aug 2016
Deep Learning with Differential Privacy Martín Abadi Andy Chu Ian Goodfellow H. B. McMahan Ilya Mironov Kunal Talwar Li Zhang FedML SyDa 216 6,155 0 01 Jul 2016
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,595 0 01 Sep 2014
Representation Learning: A Review and New Perspectives Yoshua Bengio Aaron Courville Pascal Vincent OOD SSL 274 12,458 0 24 Jun 2012