Accelerating Self-Supervised Learning via Efficient Training Strategies

11 December 2022

Mustafa Taha Koccyiugit

Timothy M. Hospedales

Hakan Bilen

SSL

ArXiv PDF HTML

Papers citing "Accelerating Self-Supervised Learning via Efficient Training Strategies"

42 / 42 papers shown

Title
CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collections Mohamed Fazli Mohamed Imam Rufael Fedaku Marew Jameel Hassan Mustansar Fiaz Alham Fikri Aji Hisham Cholakkal VLM 461 0 0 28 Nov 2024
Generative Adversarial Networks Gilad Cohen Raja Giryes GAN 273 30,109 0 01 Mar 2022
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 685 6,066 0 29 Apr 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 154 1,862 0 05 Apr 2021
EfficientNetV2: Smaller Models and Faster Training Mingxing Tan Quoc V. Le EgoV 119 2,700 0 01 Apr 2021
Barlow Twins: Self-Supervised Learning via Redundancy Reduction Jure Zbontar Li Jing Ishan Misra Yann LeCun Stéphane Deny SSL 300 2,344 0 04 Mar 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 925 29,372 0 26 Feb 2021
Big Self-Supervised Models Advance Medical Image Classification Shekoofeh Azizi Basil Mustafa Fiona Ryan Zach Beaver Jan Freyberg ... Alan Karthikesalingam Simon Kornblith Ting-Li Chen Vivek Natarajan Mohammad Norouzi SSL 112 515 0 13 Jan 2021
Exploring Simple Siamese Representation Learning Xinlei Chen Kaiming He SSL 253 4,054 0 20 Nov 2020
PyTorch Distributed: Experiences on Accelerating Data Parallel Training Shen Li Yanli Zhao R. Varma Omkar Salpekar P. Noordhuis ... Adam Paszke Jeff Smith Brian Vaughan Pritam Damania Soumith Chintala OOD MoE 55 184 0 28 Jun 2020
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin OCL SSL 230 4,074 0 17 Jun 2020
Bootstrap your own latent: A new approach to self-supervised Learning Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre Harvey Richemond ... M. G. Azar Bilal Piot Koray Kavukcuoglu Rémi Munos Michal Valko SSL 366 6,797 0 13 Jun 2020
What Makes for Good Views for Contrastive Learning? Yonglong Tian Chen Sun Ben Poole Dilip Krishnan Cordelia Schmid Phillip Isola SSL 100 1,329 0 20 May 2020
Fixing the train-test resolution discrepancy: FixEfficientNet Hugo Touvron Andrea Vedaldi Matthijs Douze Hervé Jégou AAML 243 110 0 18 Mar 2020
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 478 3,427 0 09 Mar 2020
The large learning rate phase of deep learning: the catapult mechanism Aitor Lewkowycz Yasaman Bahri Ethan Dyer Jascha Narain Sohl-Dickstein Guy Gur-Ari ODL 189 241 0 04 Mar 2020
MaxUp: A Simple Way to Improve Generalization of Neural Network Training Chengyue Gong Zhaolin Ren Mao Ye Qiang Liu AAML 65 56 0 20 Feb 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 364 18,752 0 13 Feb 2020
Momentum Contrast for Unsupervised Visual Representation Learning Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross B. Girshick SSL 199 12,074 0 13 Nov 2019
Accelerating Deep Learning by Focusing on the Biggest Losers Angela H. Jiang Daniel L.-K. Wong Giulio Zhou D. Andersen J. Dean ... Gauri Joshi M. Kaminsky M. Kozuch Zachary Chase Lipton Padmanabhan Pillai 60 121 0 02 Oct 2019
RandAugment: Practical automated data augmentation with a reduced search space E. D. Cubuk Barret Zoph Jonathon Shlens Quoc V. Le MQ 229 3,485 0 30 Sep 2019
Mix & Match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliency Elad Hoffer Berry Weinstein Itay Hubara Tal Ben-Nun Torsten Hoefler Daniel Soudry 108 20 0 12 Aug 2019
Contrastive Multiview Coding Yonglong Tian Dilip Krishnan Phillip Isola SSL 166 2,399 0 13 Jun 2019
Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination Zhirong Wu Yuanjun Xiong Stella X. Yu Dahua Lin SSL 173 3,452 0 05 May 2018
A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay L. Smith 281 1,031 0 26 Mar 2018
Unsupervised Representation Learning by Predicting Image Rotations Spyros Gidaris Praveer Singh N. Komodakis OOD SSL DRL 252 3,288 0 21 Mar 2018
Not All Samples Are Created Equal: Deep Learning with Importance Sampling Angelos Katharopoulos François Fleuret 89 519 0 02 Mar 2018
Three Factors Influencing Minima in SGD Stanislaw Jastrzebski Zachary Kenton Devansh Arpit Nicolas Ballas Asja Fischer Yoshua Bengio Amos Storkey 76 463 0 13 Nov 2017
Don't Decay the Learning Rate, Increase the Batch Size Samuel L. Smith Pieter-Jan Kindermans Chris Ying Quoc V. Le ODL 99 995 0 01 Nov 2017
Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks Pratik Chaudhari Stefano Soatto MLT 68 304 0 30 Oct 2017
Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates L. Smith Nicholay Topin AI4CE 84 520 0 23 Aug 2017
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour Priya Goyal Piotr Dollár Ross B. Girshick P. Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia Kaiming He 3DH 126 3,678 0 08 Jun 2017
Colorization as a Proxy Task for Visual Understanding Gustav Larsson Michael Maire Gregory Shakhnarovich SSL 145 497 0 11 Mar 2017
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction Richard Y. Zhang Phillip Isola Alexei A. Efros SSL DRL 63 668 0 29 Nov 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 424 2,937 0 15 Sep 2016
Training Region-based Object Detectors with Online Hard Example Mining Abhinav Shrivastava Abhinav Gupta Ross B. Girshick ObjD 148 2,418 0 12 Apr 2016
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles M. Noroozi Paolo Favaro SSL 171 2,979 0 30 Mar 2016
Colorful Image Colorization Richard Y. Zhang Phillip Isola Alexei A. Efros 127 3,529 0 28 Mar 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 193,878 0 10 Dec 2015
Cyclical Learning Rates for Training Neural Networks L. Smith ODL 197 2,525 0 03 Jun 2015
Unsupervised Visual Representation Learning by Context Prediction Carl Doersch Abhinav Gupta Alexei A. Efros DRL SSL 164 2,781 0 19 May 2015
Auto-Encoding Variational Bayes Diederik P. Kingma Max Welling BDL 452 16,933 0 20 Dec 2013