Scaling Backwards: Minimal Synthetic Pre-training?

Scaling Backwards: Minimal Synthetic Pre-training?

1 August 2024

Ryu Tadokoro

Christian Rupprecht

Hirokatsu Kataoka

Papers citing "Scaling Backwards: Minimal Synthetic Pre-training?"

12 / 12 papers shown

Title
Formula-Supervised Sound Event Detection: Pre-Training Without Real Data Yuto Shibata Keitaro Tanaka Yoshiaki Bando Keisuke Imoto Hirokatsu Kataoka Yoshimitsu Aoki 36 0 0 06 Apr 2025
MoireDB: Formula-generated Interference-fringe Image Dataset Yuto Matsuo Ryo Hayamizu Hirokatsu Kataoka Akio Nakamura 47 0 0 03 Feb 2025
A Vision Check-up for Language Models Pratyusha Sharma Tamar Rott Shaham Manel Baradad Stephanie Fu Adrian Rodriguez-Munoz Shivam Duggal Phillip Isola Antonio Torralba VLM LRM 78 24 0 03 Jan 2024
Visual Atoms: Pre-training Vision Transformers with Sinusoidal Waves Sora Takashima Ryo Hayamizu Nakamasa Inoue Hirokatsu Kataoka Rio Yokota 68 20 0 02 Mar 2023
Training Vision Transformers with Only 2040 Images Yunhao Cao Hao Yu Jianxin Wu ViT 120 43 0 26 Jan 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 322 7,503 0 11 Nov 2021
Improving Fractal Pre-training Connor Anderson Ryan Farrell 95 27 0 06 Oct 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 389 5,832 0 29 Apr 2021
Pre-training without Natural Images Hirokatsu Kataoka Kazushige Okayasu Asato Matsumoto Eisuke Yamagata Ryosuke Yamada Nakamasa Inoue Akio Nakamura Y. Satoh 81 117 0 21 Jan 2021
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 281 3,386 0 09 Mar 2020
Boosting Self-Supervised Learning via Knowledge Transfer M. Noroozi Ananth Vinjimoor Paolo Favaro Hamed Pirsiavash SSL 224 292 0 01 May 2018
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 341 39,252 0 01 Sep 2014