Deduplicating Training Data Makes Language Models Better

14 July 2021

Papers citing "Deduplicating Training Data Makes Language Models Better"

25 / 75 papers shown

Title
Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus Jack Bandy Nicholas Vincent 50 57 0 11 May 2021
PanGu- $α$ : Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation Wei Zeng Xiaozhe Ren Teng Su Hui Wang Yi-Lun Liao ... Gaojun Fan Yaowei Wang Xuefeng Jin Qun Liu Yonghong Tian ALM MoE AI4CE 67 213 0 26 Apr 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 321 658 0 21 Apr 2021
Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifiers Piotr Teterwak Chiyuan Zhang Dilip Krishnan Michael C. Mozer 46 9 0 15 Mar 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 388 2,051 0 31 Dec 2020
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 415 1,868 0 14 Dec 2020
mT5: A massively multilingual pre-trained text-to-text transformer Linting Xue Noah Constant Adam Roberts Mihir Kale Rami Al-Rfou Aditya Siddhant Aditya Barua Colin Raffel 98 2,489 0 22 Oct 2020
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation Vitaly Feldman Chiyuan Zhang TDI 100 454 0 09 Aug 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 512 41,106 0 28 May 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 298 19,824 0 23 Oct 2019
Universal Adversarial Triggers for Attacking and Analyzing NLP Eric Wallace Shi Feng Nikhil Kandpal Matt Gardner Sameer Singh AAML SILM 92 856 0 20 Aug 2019
Energy and Policy Considerations for Deep Learning in NLP Emma Strubell Ananya Ganesh Andrew McCallum 53 2,633 0 05 Jun 2019
Defending Against Neural Fake News Rowan Zellers Ari Holtzman Hannah Rashkin Yonatan Bisk Ali Farhadi Franziska Roesner Yejin Choi AAML 102 1,019 0 29 May 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 144 3,714 0 09 Jan 2019
Detecting Overfitting of Deep Generative Networks via Latent Recovery Ryan Webster Julien Rabin Loïc Simon F. Jurie GAN 36 99 0 09 Jan 2019
The Adverse Effects of Code Duplication in Machine Learning Models of Code Miltiadis Allamanis 58 319 0 16 Dec 2018
Connected Components at Scale via Local Contractions Jakub Lacki Vahab Mirrokni Michał Włodarczyk 29 18 0 27 Jul 2018
A Simple Method for Commonsense Reasoning Trieu H. Trinh Quoc V. Le LRM ReLM 85 432 0 07 Jun 2018
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost Noam M. Shazeer Mitchell Stern ODL 56 1,032 0 11 Apr 2018
Datasheets for Datasets Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer Wortman Vaughan Hanna M. Wallach Hal Daumé Kate Crawford 219 2,158 0 23 Mar 2018
A Closer Look at Memorization in Deep Networks Devansh Arpit Stanislaw Jastrzebski Nicolas Ballas David M. Krueger Emmanuel Bengio ... Tegan Maharaj Asja Fischer Aaron Courville Yoshua Bengio Simon Lacoste-Julien TDI 107 1,801 0 16 Jun 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 492 129,831 0 12 Jun 2017
Membership Inference Attacks against Machine Learning Models Reza Shokri M. Stronati Congzheng Song Vitaly Shmatikov SLR MIALM MIACV 209 4,075 0 18 Oct 2016
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books Yukun Zhu Ryan Kiros R. Zemel Ruslan Salakhutdinov R. Urtasun Antonio Torralba Sanja Fidler 110 2,529 0 22 Jun 2015
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling Ciprian Chelba Tomas Mikolov M. Schuster Qi Ge T. Brants P. Koehn T. Robinson 121 1,099 0 11 Dec 2013