The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

1 June 2023

Guilherme Penedo

Quentin Malartic

Daniel Hesslow

Ruxandra-Aimée Cojocaru

Papers citing "The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only"

37 / 587 papers shown

Title
On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook Mingyuan Fan Chengyu Wang Cen Chen Yang Liu Jun Huang HILM 39 3 0 31 Jul 2023
NLLG Quarterly arXiv Report 06/23: What are the most influential current AI Papers? Steffen Eger Christoph Leiter Jonas Belouadi Ran Zhang Aida Kostikova Daniil Larionov Yanran Chen Vivian Fresen AI4CE 29 4 0 31 Jul 2023
Med-HALT: Medical Domain Hallucination Test for Large Language Models Ankit Pal Logesh Kumar Umapathi Malaikannan Sankarasubbu HILM LM&MA VLM 36 128 0 28 Jul 2023
Universal and Transferable Adversarial Attacks on Aligned Language Models Andy Zou Zifan Wang Nicholas Carlini Milad Nasr J. Zico Kolter Matt Fredrikson 92 1,278 0 27 Jul 2023
General Purpose Artificial Intelligence Systems (GPAIS): Properties, Definition, Taxonomy, Societal Implications and Responsible Governance I. Triguero Daniel Molina Javier Poyatos Javier Del Ser Francisco Herrera AI4TS AI4MH 34 5 0 26 Jul 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 120 11,099 0 18 Jul 2023
Generating Benchmarks for Factuality Evaluation of Language Models Dor Muhlgay Ori Ram Inbal Magar Yoav Levine Nir Ratner Yonatan Belinkov Omri Abend Kevin Leyton-Brown Amnon Shashua Y. Shoham HILM 33 91 0 13 Jul 2023
PolyLM: An Open Source Polyglot Large Language Model Xiangpeng Wei Hao-Ran Wei Huan Lin Tianhao Li Pei Zhang ... Yu Bowen Dayiheng Liu Baosong Yang Fei Huang Jun Xie LRM 48 55 0 12 Jul 2023
MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problems Markus J. Buehler AI4CE 29 41 0 30 Jun 2023
Large Multimodal Models: Notes on CVPR 2023 Tutorial Chunyuan Li MLLM VLM 19 20 0 26 Jun 2023
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models Shizhe Diao Rui Pan Hanze Dong Kashun Shum Jipeng Zhang Wei Xiong Tong Zhang ALM 22 63 0 21 Jun 2023
CMMLU: Measuring massive multitask language understanding in Chinese Haonan Li Yixuan Zhang Fajri Koto Yifei Yang Hai Zhao Yeyun Gong Nan Duan Tim Baldwin ALM ELM 47 239 0 15 Jun 2023
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration Ji Lin Jiaming Tang Haotian Tang Shang Yang Wei-Ming Chen Wei-Chen Wang Guangxuan Xiao Xingyu Dang Chuang Gan Song Han EDL MQ 36 474 0 01 Jun 2023
Scaling Data-Constrained Language Models Niklas Muennighoff Alexander M. Rush Boaz Barak Teven Le Scao Aleksandra Piktus Nouamane Tazi S. Pyysalo Thomas Wolf Colin Raffel ALM 38 200 0 25 May 2023
Multilingual Large Language Models Are Not (Yet) Code-Switchers Ruochen Zhang Samuel Cahyawijaya Jan Christian Blaise Cruz Genta Indra Winata Alham Fikri Aji LRM 33 52 0 23 May 2023
Exploring Self-supervised Logic-enhanced Training for Large Language Models Fangkai Jiao Zhiyang Teng Bosheng Ding Zhengyuan Liu Nancy F. Chen Chenyu You ReLM LRM 32 4 0 23 May 2023
Towards Legally Enforceable Hate Speech Detection for Public Forums Chunyan Luo R. Bhambhoria Xiao-Dan Zhu Samuel Dahan AILaw 33 5 0 23 May 2023
Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models Tim Schott Daniel Furman Shreshta Bhat ELM 35 4 0 23 May 2023
Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistance Yue Zhang Leyang Cui Deng Cai Xinting Huang Tao Fang Wei Bi ALM 29 36 0 22 May 2023
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity Shayne Longpre Gregory Yauney Emily Reif Katherine Lee Adam Roberts ... Denny Zhou Jason W. Wei Kevin Robinson David M. Mimno Daphne Ippolito 29 149 0 22 May 2023
Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach Masahiro Kaneko Graham Neubig Naoaki Okazaki 39 6 0 19 May 2023
Statistical Knowledge Assessment for Large Language Models Qingxiu Dong Jingjing Xu Lingpeng Kong Zhifang Sui Lei Li HILM 47 6 0 17 May 2023
The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks Anders Giovanni Møller Jacob Aarup Dalsgaard Arianna Pera L. Aiello 81 35 0 26 Apr 2023
nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales Yiqun Yao Siqi Fan Xiusheng Huang Xuezhi Fang Xiang Li ... Peng Han Shuo Shang Kang Liu Aixin Sun Yequan Wang 33 6 0 14 Apr 2023
Querying Large Language Models with SQL Mohammed Saeed Nicola De Cao Paolo Papotti 27 29 0 02 Apr 2023
Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering Zhou Yu Xuecheng Ouyang Zhenwei Shao Mei Wang Jun Yu MLLM 94 11 0 03 Mar 2023
Poisoning Web-Scale Training Datasets is Practical Nicholas Carlini Matthew Jagielski Christopher A. Choquette-Choo Daniel Paleka Will Pearce Hyrum S. Anderson Andreas Terzis Kurt Thomas Florian Tramèr SILM 31 182 0 20 Feb 2023
JASMINE: Arabic GPT Models for Few-Shot Learning El Moatez Billah Nagoudi Muhammad Abdul-Mageed AbdelRahim Elmadany Alcides Alcoba Inciarte Md. Tawkat Islam Khondaker 33 7 0 21 Dec 2022
GPT-Neo for commonsense reasoning -- a theoretical and practical lens Rohan Kashyap Vivek Kashyap Narendra C.P ReLM ELM LRM 38 7 0 28 Nov 2022
What Language Model to Train if You Have One Million GPU Hours? Teven Le Scao Thomas Wang Daniel Hesslow Lucile Saulnier Stas Bekman ... Lintang Sutawika Jaesung Tae Zheng-Xin Yong Julien Launay Iz Beltagy MoE AI4CE 230 103 0 27 Oct 2022
Challenges in Detoxifying Language Models Johannes Welbl Amelia Glaese J. Uesato Sumanth Dathathri John F. J. Mellor Lisa Anne Hendricks Kirsty Anderson Pushmeet Kohli Ben Coppin Po-Sen Huang LM&MA 250 193 0 15 Sep 2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 253 698 0 27 Aug 2021
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 242 593 0 14 Jul 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 282 1,996 0 31 Dec 2020
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 290 1,824 0 14 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,489 0 23 Jan 2020
PubMedQA: A Dataset for Biomedical Research Question Answering Qiao Jin Bhuwan Dhingra Zhengping Liu William W. Cohen Xinghua Lu 243 815 0 13 Sep 2019