The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

1 June 2023

Guilherme Penedo

Quentin Malartic

Daniel Hesslow

Ruxandra-Aimée Cojocaru

Papers citing "The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only"

50 / 587 papers shown

Title
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models L. Yu Weisen Jiang Han Shi Jincheng Yu Zhengying Liu Yu Zhang James T. Kwok Zheng Li Adrian Weller Weiyang Liu OSLM LRM 50 337 0 21 Sep 2023
BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model Nolan Dey Daria Soboleva Faisal Al-Khateeb Bowen Yang Ribhu Pathria ... Robert Myers Jacob Robert Steeves Natalia Vassilieva Marvin Tom Joel Hestness MoE 27 15 0 20 Sep 2023
DreamLLM: Synergistic Multimodal Comprehension and Creation Runpei Dong Chunrui Han Yuang Peng Zekun Qi Zheng Ge ... Hao-Ran Wei Xiangwen Kong Xiangyu Zhang Kaisheng Ma Li Yi MLLM 39 173 0 20 Sep 2023
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data Guan-Bo Wang Sijie Cheng Xianyuan Zhan Xiangang Li Sen Song Yang Liu ALM 27 231 0 20 Sep 2023
SlimPajama-DC: Understanding Data Combinations for LLM Training Zhiqiang Shen Tianhua Tao Liqun Ma W. Neiswanger Zhengzhong Liu ... Bowen Tan Joel Hestness Natalia Vassilieva Daria Soboleva Eric P. Xing 27 45 0 19 Sep 2023
OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratch Juntao Li Zecheng Tang Yuyang Ding Pinzheng Wang Pei Guo ... Wenliang Chen Guohong Fu Qiaoming Zhu Guodong Zhou Hao Fei 45 5 0 19 Sep 2023
Baichuan 2: Open Large-scale Language Models Ai Ming Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian ... Youxin Jiang Yuchen Gao Yupeng Zhang Zenan Zhou Zhiying Wu ELM LRM 75 703 0 19 Sep 2023
CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages Thuat Nguyen Chien Van Nguyen Viet Dac Lai Hieu Man Nghia Trung Ngo Franck Dernoncourt Ryan A. Rossi Thien Huu Nguyen 45 97 0 17 Sep 2023
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference Parsa Kavehzadeh Mojtaba Valipour Marzieh S. Tahaei Ali Ghodsi Boxing Chen Mehdi Rezagholizadeh 35 6 0 16 Sep 2023
Neural Machine Translation Models Can Learn to be Few-shot Learners Raphael Reinauer P. Simianer Kaden Uhlig Johannes E. M. Mosig Joern Wuebker LRM 29 8 0 15 Sep 2023
ICLEF: In-Context Learning with Expert Feedback for Explainable Style Transfer Arkadiy Saakyan Smaranda Muresan 26 3 0 15 Sep 2023
Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suite Chan-Jan Hsu Chang-Le Liu Feng-Ting Liao Po-Chun Hsu Yi-Chang Chen Da-shan Shiu ELM ALM 22 12 0 15 Sep 2023
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions Federico Bianchi Mirac Suzgun Giuseppe Attanasio Paul Röttger Dan Jurafsky Tatsunori Hashimoto James Zou ALM LM&MA LRM 34 181 0 14 Sep 2023
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning Xiang Yue Xingwei Qu Ge Zhang Yao Fu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen AIMat LRM 64 368 0 11 Sep 2023
Effective Proxy for Human Labeling: Ensemble Disagreement Scores in Large Language Models for Industrial NLP Wei Du Laksh Advani Yashmeet Gambhir Daniel J. Perry Prashant Shiralkar Zhengzheng Xing Aaron Colak ALM 35 1 0 11 Sep 2023
An Empirical Study of NetOps Capability of Pre-Trained Large Language Models Yukai Miao Yu Bai Li Chen Dan Li Haifeng Sun ... Dapeng Sun Xiuting Xu Qi Zhang Chao Xiang Xinchi Li ELM 11 10 0 11 Sep 2023
Textbooks Are All You Need II: phi-1.5 technical report Yuan-Fang Li Sébastien Bubeck Ronen Eldan Allison Del Giorno Suriya Gunasekar Yin Tat Lee ALM LRM 36 445 0 11 Sep 2023
Understanding the Impact of Post-Training Quantization on Large Language Models Somnath Roy MQ 38 3 0 11 Sep 2023
HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models Guijin Son Hanwool Albert Lee Suwan Kim Huiseo Kim Jaecheol Lee Je Won Yeom Jihyu Jung Jung Woo Kim Songseong Kim RALM ELM 34 20 0 06 Sep 2023
Data-Juicer: A One-Stop Data Processing System for Large Language Models Daoyuan Chen Yilun Huang Zhijian Ma Hesen Chen Xuchen Pan ... Zhaoyang Liu Jinyang Gao Yaliang Li Bolin Ding Jingren Zhou SyDa VLM 31 30 0 05 Sep 2023
QuantEase: Optimization-based Quantization for Language Models Kayhan Behdin Ayan Acharya Aman Gupta Qingquan Song Siyu Zhu S. Keerthi Rahul Mazumder MQ 30 20 0 05 Sep 2023
Unveiling Theory of Mind in Large Language Models: A Parallel to Single Neurons in the Human Brain Mohsen Jamali Ziv M. Williams Jing Cai 24 19 0 04 Sep 2023
Open Sesame! Universal Black Box Jailbreaking of Large Language Models Raz Lapid Ron Langberg Moshe Sipper AAML 27 105 0 04 Sep 2023
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models Yue Zhang Yafu Li Leyang Cui Deng Cai Lemao Liu ... Longyue Wang A. Luu Wei Bi Freda Shi Shuming Shi RALM LRM HILM 48 522 0 03 Sep 2023
Baseline Defenses for Adversarial Attacks Against Aligned Language Models Neel Jain Avi Schwarzschild Yuxin Wen Gowthami Somepalli John Kirchenbauer Ping Yeh-Chiang Micah Goldblum Aniruddha Saha Jonas Geiping Tom Goldstein AAML 60 340 0 01 Sep 2023
No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Function Haotian Xu LRM 38 12 0 01 Sep 2023
Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior Ashmit Khandelwal Aditya Agrawal Aanisha Bhattacharyya Yaman Kumar Singla Somesh Singh ... Ishita Dasgupta Stefano Petrangeli R. Shah Changyou Chen Balaji Krishnamurthy 24 8 0 01 Sep 2023
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants Lucas Bandarkar Davis Liang Benjamin Muller Mikel Artetxe Satya Narayan Shukla Don Husa Naman Goyal Abhinandan Krishnan Luke Zettlemoyer Madian Khabsa 30 133 0 31 Aug 2023
Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models Neha Sengupta Sunil Kumar Sahu Bokang Jia Satheesh Katipomu Haonan Li ... A. Jackson Hector Xuguang Ren Preslav Nakov Timothy Baldwin Eric P. Xing LRM 29 40 0 30 Aug 2023
Fine-Tuning Llama 2 Large Language Models for Detecting Online Sexual Predatory Chats and Abusive Texts Thanh Thi Nguyen Campbell Wilson Janis Dalins 27 23 0 28 Aug 2023
Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrations Leonardo Ranaldi Giulia Pucci André Freitas 35 33 0 27 Aug 2023
Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models Kaiyuan Gao Su He Zhenyu He Jiacheng Lin Qizhi Pei Jie Shao Wei Zhang LM&MA SyDa 38 4 0 27 Aug 2023
The Poison of Alignment Aibek Bekbayev Sungbae Chun Yerzat Dulat James Yamazaki 28 9 0 25 Aug 2023
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models Wenqi Shao Yonghong Tian Zhaoyang Zhang Peng Xu Lirui Zhao Zhiqiang Li Kaipeng Zhang Peng Gao Yu Qiao Ping Luo MQ 30 178 0 25 Aug 2023
Code Llama: Open Foundation Models for Code Baptiste Rozière Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat ... Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom Gabriel Synnaeve ELM ALM 63 1,906 0 24 Aug 2023
CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias Vipul Gupta Pranav Narayanan Venkit Hugo Laurenccon Shomir Wilson R. Passonneau 46 12 0 24 Aug 2023
Considerations for health care institutions training large language models on electronic health records Weipeng Zhou Danielle Bitterman Majid Afshar Timothy A. Miller LM&MA 31 0 0 24 Aug 2023
D4: Improving LLM Pretraining via Document De-Duplication and Diversification Kushal Tirumala Daniel Simig Armen Aghajanyan Ari S. Morcos SyDa 13 104 0 23 Aug 2023
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning Ming Li Yong Zhang Zhitao Li Jiuhai Chen Lichang Chen Ning Cheng Jianzong Wang Dinesh Manocha Jing Xiao 45 176 0 23 Aug 2023
Cabrita: closing the gap for foreign languages Celio H. N. Larcher Marcos Piau Paulo Finardi P. Gengo P. Esposito Vinicius Fernandes Caridá CLL 21 19 0 23 Aug 2023
Instruction Tuning for Large Language Models: A Survey Shengyu Zhang Linfeng Dong Xiaoya Li Sen Zhang Xiaofei Sun ... Jiwei Li Runyi Hu Tianwei Zhang Fei Wu Guoyin Wang LM&MA 24 546 0 21 Aug 2023
FootGPT : A Large Language Model Development Experiment on a Minimal Setting Eren Unlu ALM 24 0 0 16 Aug 2023
Time Travel in LLMs: Tracing Data Contamination in Large Language Models Shahriar Golchin Mihai Surdeanu 35 93 0 16 Aug 2023
EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce Yong Li Shirong Ma Xiaobin Wang Shen Huang Chengyue Jiang Haitao Zheng Pengjun Xie Fei Huang Yong-jia Jiang RALM ALM LRM 34 49 0 14 Aug 2023
InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models Kabir Nagrecha Lingyi Liu P. Delgado Prasanna Padmanabhan OffRL AI4CE 33 5 0 13 Aug 2023
Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT Jing Yang Cong Liu Wendy Deng Dangwei Wu C. Weng Yunyun Zhou Kai Wang 27 20 0 11 Aug 2023
A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology Sean Wu Michael Koo L. Blum A. Black Liyo Kao Fabien Scalzo Ira Kurtz LM&MA ELM AI4MH 26 42 0 09 Aug 2023
Causality Guided Disentanglement for Cross-Platform Hate Speech Detection Paras Sheth Tharindu Kumarage Raha Moraffah Amanat Chadha Huan Liu 31 7 0 03 Aug 2023
Adapt and Decompose: Efficient Generalization of Text-to-SQL via Domain Adapted Least-To-Most Prompting Aseem Arora Shabbirhussain Bhaisaheb Harshit Nigam Manasi S. Patwardhan L. Vig Gautam M. Shroff 27 8 0 01 Aug 2023
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs Yujia Qin Shi Liang Yining Ye Kunlun Zhu Lan Yan ... Jie Zhou Mark B. Gerstein Dahai Li Zhiyuan Liu Maosong Sun CLL ALM LLMAG ELM LM&MA 87 628 0 31 Jul 2023