On Catastrophic Inheritance of Large Foundation Models

On Catastrophic Inheritance of Large Foundation Models

2 February 2024

Hao Chen

Bhiksha Raj

Xing Xie

ArXiv (abs)PDF HTML

Papers citing "On Catastrophic Inheritance of Large Foundation Models"

16 / 66 papers shown

Title
What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus A. Luccioni J. Viviano 74 118 0 06 May 2021
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus Jesse Dodge Maarten Sap Ana Marasović William Agnew Gabriel Ilharco Dirk Groeneveld Margaret Mitchell Matt Gardner AILaw 118 448 0 18 Apr 2021
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets Julia Kreutzer Isaac Caswell Lisa Wang Ahsan Wahab D. Esch ... Duygu Ataman Orevaoghene Ahia Oghenefego Ahia Sweta Agrawal Mofetoluwa Adeyemi 56 278 0 22 Mar 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 458 2,120 0 31 Dec 2020
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation Vitaly Feldman Chiyuan Zhang TDI 205 470 0 09 Aug 2020
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization Dan Hendrycks Steven Basart Norman Mu Saurav Kadavath Frank Wang ... Samyak Parajuli Mike Guo Basel Alomair Jacob Steinhardt Justin Gilmer OOD 350 1,754 0 29 Jun 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 611 4,905 0 23 Jan 2020
Big Transfer (BiT): General Visual Representation Learning Alexander Kolesnikov Lucas Beyer Xiaohua Zhai J. Puigcerver Jessica Yung Sylvain Gelly N. Houlsby MQ 288 1,211 0 24 Dec 2019
PyHessian: Neural Networks Through the Lens of the Hessian Z. Yao A. Gholami Kurt Keutzer Michael W. Mahoney ODL 63 303 0 16 Dec 2019
A Comprehensive Survey on Transfer Learning Fuzhen Zhuang Zhiyuan Qi Keyu Duan Dongbo Xi Yongchun Zhu Hengshu Zhu Hui Xiong Qing He 186 4,471 0 07 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 470 20,317 0 23 Oct 2019
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian Samet Oymak Zalan Fabian Mingchen Li Mahdi Soltanolkotabi MLT 66 88 0 12 Jun 2019
Does Learning Require Memorization? A Short Tale about a Long Tail Vitaly Feldman TDI 132 498 0 12 Jun 2019
Do ImageNet Classifiers Generalize to ImageNet? Benjamin Recht Rebecca Roelofs Ludwig Schmidt Vaishaal Shankar OOD SSeg VLM 121 1,726 0 13 Feb 2019
Generalization in Deep Learning Kenji Kawaguchi L. Kaelbling Yoshua Bengio ODL 99 460 0 16 Oct 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 348 4,635 0 10 Nov 2016