Downstream Datasets Make Surprisingly Good Pretraining Corpora

Downstream Datasets Make Surprisingly Good Pretraining Corpora

28 September 2022

Jeffrey P. Bigham

Zachary Chase Lipton

Papers citing "Downstream Datasets Make Surprisingly Good Pretraining Corpora"

11 / 11 papers shown

Title
Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? Ahmed Alajrami Katerina Margatina Nikolaos Aletras AAML 19 1 0 26 Oct 2023
Utility Theory of Synthetic Data Generation Shi Xu W. Sun Guang Cheng 25 5 0 17 May 2023
How to Train Your CheXDragon: Training Chest X-Ray Models for Transfer to Novel Tasks and Healthcare Systems Cara Van Uden Jeremy Irvin Mars Huang N. Dean J. Carr A. Ng C. Langlotz OOD 24 1 0 13 May 2023
On the Role of Parallel Data in Cross-lingual Transfer Learning Machel Reid Mikel Artetxe 21 10 0 20 Dec 2022
Blessing of Class Diversity in Pre-training Yulai Zhao Jianshu Chen S. Du AI4CE 10 3 0 07 Sep 2022
Can Wikipedia Help Offline Reinforcement Learning? Machel Reid Yutaro Yamada S. Gu 3DV RALM OffRL 137 95 0 28 Jan 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 305 7,443 0 11 Nov 2021
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework Xingcheng Yao Yanan Zheng Xiaocong Yang Zhilin Yang 30 44 0 07 Nov 2021
Mitigating Language-Dependent Ethnic Bias in BERT Jaimeen Ahn Alice H. Oh 139 91 0 13 Sep 2021
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 415 2,586 0 03 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018