The Role of Pre-training Data in Transfer Learning

27 February 2023

Papers citing "The Role of Pre-training Data in Transfer Learning"

31 / 31 papers shown

Title
Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP Thao Nguyen Gabriel Ilharco Mitchell Wortsman Sewoong Oh Ludwig Schmidt CLIP VLM 104 104 0 10 Aug 2022
Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning Shibani Santurkar Yann Dubois Rohan Taori Percy Liang Tatsunori Hashimoto CLIP VLM 57 41 0 15 Jul 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 382 3,542 0 29 Apr 2022
A Broad Study of Pre-training for Domain Generalization and Adaptation Donghyun Kim Kaihong Wang Stan Sclaroff Kate Saenko OOD AI4CE 70 80 0 22 Mar 2022
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution Ananya Kumar Aditi Raghunathan Robbie Jones Tengyu Ma Percy Liang OODD 114 674 0 21 Feb 2022
SLIP: Self-supervision meets Language-Image Pre-training Norman Mu Alexander Kirillov David Wagner Saining Xie VLM CLIP 141 488 0 23 Dec 2021
RedCaps: web-curated image-text data created by the people, for the people Karan Desai Gaurav Kaul Zubin Aysola Justin Johnson 116 166 0 22 Nov 2021
Combined Scaling for Zero-shot Transfer Learning Hieu H. Pham Zihang Dai Golnaz Ghiasi Kenji Kawaguchi Hanxiao Liu ... Yi-Ting Chen Minh-Thang Luong Yonghui Wu Mingxing Tan Quoc V. Le VLM 63 199 0 19 Nov 2021
LiT: Zero-Shot Transfer with Locked-image text Tuning Xiaohua Zhai Tianlin Li Basil Mustafa Andreas Steiner Daniel Keysers Alexander Kolesnikov Lucas Beyer VLM 95 556 0 15 Nov 2021
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs Christoph Schuhmann Richard Vencu Romain Beaumont R. Kaczmarczyk Clayton Mullis Aarush Katta Theo Coombes J. Jitsev Aran Komatsuzaki VLM MLLM CLIP 223 1,428 0 03 Nov 2021
Exploring the Limits of Large Scale Pre-training Samira Abnar Mostafa Dehghani Behnam Neyshabur Hanie Sedghi AI4CE 91 118 0 05 Oct 2021
Robust fine-tuning of zero-shot models Mitchell Wortsman Gabriel Ilharco Jong Wook Kim Mike Li Simon Kornblith ... Raphael Gontijo-Lopes Hannaneh Hajishirzi Ali Farhadi Hongseok Namkoong Ludwig Schmidt VLM 129 724 0 04 Sep 2021
Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization John Miller Rohan Taori Aditi Raghunathan Shiori Sagawa Pang Wei Koh Vaishaal Shankar Percy Liang Y. Carmon Ludwig Schmidt OODD OOD 74 278 0 09 Jul 2021
A Broad Study on the Transferability of Visual Representations with Contrastive Learning Ashraful Islam Chun-Fu Chen Yikang Shen Leonid Karlinsky Richard J. Radke Rogerio Feris SSL 118 113 0 24 Mar 2021
Self-supervised Pretraining of Visual Features in the Wild Priya Goyal Mathilde Caron Benjamin Lefaudeux Min Xu Pengchao Wang ... Mannat Singh Vitaliy Liptchinsky Ishan Misra Armand Joulin Piotr Bojanowski VLM SSL 74 274 0 02 Mar 2021
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning Krishna Srinivasan K. Raman Jiecao Chen Michael Bendersky Marc Najork VLM 258 316 0 02 Mar 2021
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts Soravit Changpinyo P. Sharma Nan Ding Radu Soricut VLM 432 1,127 0 17 Feb 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 443 3,842 0 11 Feb 2021
A linearized framework and a new benchmark for model selection for fine-tuning Aditya Deshpande Alessandro Achille Avinash Ravichandran Hao Li Luca Zancato Charless C. Fowlkes Rahul Bhotika Stefano Soatto Pietro Perona ALM 154 47 0 29 Jan 2021
How Well Do Self-Supervised Models Transfer? Linus Ericsson Henry Gouk Timothy M. Hospedales SSL 110 278 0 26 Nov 2020
What is being transferred in transfer learning? Behnam Neyshabur Hanie Sedghi Chiyuan Zhang 103 522 0 26 Aug 2020
On Robustness and Transferability of Convolutional Neural Networks Josip Djolonga Jessica Yung Michael Tschannen Rob Romijnders Lucas Beyer ... D. Moldovan Sylvain Gelly N. Houlsby Xiaohua Zhai Mario Lucic OOD 62 155 0 16 Jul 2020
Measuring Robustness to Natural Distribution Shifts in Image Classification Rohan Taori Achal Dave Vaishaal Shankar Nicholas Carlini Benjamin Recht Ludwig Schmidt OOD 113 546 0 01 Jul 2020
LEEP: A New Measure to Evaluate Transferability of Learned Representations Cuong V Nguyen Tal Hassner Matthias Seeger Cédric Archambeau 74 217 0 27 Feb 2020
Transfusion: Understanding Transfer Learning for Medical Imaging M. Raghu Chiyuan Zhang Jon M. Kleinberg Samy Bengio MedIm 77 984 0 14 Feb 2019
Do Better ImageNet Models Transfer Better? Simon Kornblith Jonathon Shlens Quoc V. Le OOD MLT 158 1,327 0 23 May 2018
EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification P. Helber B. Bischke Andreas Dengel Damian Borth 133 1,817 0 31 Aug 2017
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era Chen Sun Abhinav Shrivastava Saurabh Singh Abhinav Gupta VLM 185 2,398 0 10 Jul 2017
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 330 8,116 0 13 Aug 2016
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,525 0 01 Sep 2014
Describing Textures in the Wild Mircea Cimpoi Subhransu Maji Iasonas Kokkinos S. Mohamed Andrea Vedaldi 3DV 119 2,671 0 14 Nov 2013