The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

1 June 2023

Guilherme Penedo

Quentin Malartic

Daniel Hesslow

Ruxandra-Aimée Cojocaru

Papers citing "The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only"

50 / 131 papers shown

Title
Small Language Models: Survey, Measurements, and Insights Zhenyan Lu Xiang Li Dongqi Cai Rongjie Yi Fangming Liu Xiwen Zhang Nicholas D. Lane Mengwei Xu ObjD LRM 157 58 0 24 Sep 2024
Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review Neha Prakriya Jui-Nan Yen Cho-Jui Hsieh Jason Cong KELM AI4CE LRM 105 1 0 10 Sep 2024
What is the Role of Small Models in the LLM Era: A Survey Lihu Chen Gaël Varoquaux ALM 239 32 0 10 Sep 2024
Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost Sania Nayab Giulio Rossolini Giorgio Buttazzo Nicolamaria Manes F. Giacomelli Nicolamaria Manes Fabrizio Giacomelli LRM 133 42 0 29 Jul 2024
INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages A. Singh Rudra Murthy Vishwajeet Kumar Jaydeep Sen Ashish Mittal Ganesh Ramakrishnan 201 6 0 18 Jul 2024
Scaling Retrieval-Based Language Models with a Trillion-Token Datastore Rulin Shao Jacqueline He Akari Asai Weijia Shi Tim Dettmers Sewon Min Luke Zettlemoyer Pang Wei Koh RALM 94 26 0 09 Jul 2024
Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model Longrong Yang Dong Shen Chaoxiang Cai Fan Yang Size Li Tingting Gao Xi Li MoE 144 2 0 28 Jun 2024
Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization Yuanpu Cao Tianrong Zhang Bochuan Cao Ziyi Yin Lu Lin Fenglong Ma Jinghui Chen LLMSV 88 33 0 28 May 2024
Multi-Reference Preference Optimization for Large Language Models Hung Le Quan Tran D. Nguyen Kien Do Saloni Mittal Kelechi Ogueji Svetha Venkatesh 82 1 0 26 May 2024
Linearizing Large Language Models Jean Mercat Igor Vasiljevic Sedrick Scott Keh Kushal Arora Achal Dave Adrien Gaidon Thomas Kollar 101 24 0 10 May 2024
Large Language Models for Cyber Security: A Systematic Literature Review HanXiang Xu Shenao Wang Ningke Li Kaidi Wang Yanjie Zhao Kai Chen Ting Yu Yang Liu Haoyu Wang 117 43 0 08 May 2024
Hallucination of Multimodal Large Language Models: A Survey Zechen Bai Pichao Wang Tianjun Xiao Tong He Zongbo Han Zheng Zhang Mike Zheng Shou VLM LRM 251 197 0 29 Apr 2024
Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities Kazuki Fujii Taishi Nakamura Mengsay Loem Hiroki Iida Masanari Ohi Kakeru Hattori Hirai Shota Sakae Mizuki Rio Yokota Naoaki Okazaki CLL 123 73 0 27 Apr 2024
CORM: Cache Optimization with Recent Message for Large Language Model Inference Jincheng Dai Zhuowei Huang Haiyun Jiang Chen Chen Deng Cai Wei Bi Shuming Shi 106 3 0 24 Apr 2024
AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs Anselm Paulus Arman Zharmagambetov Chuan Guo Brandon Amos Yuandong Tian AAML 142 67 0 21 Apr 2024
HLAT: High-quality Large Language Model Pre-trained on AWS Trainium Haozheng Fan Hao Zhou Guangtai Huang Parameswaran Raman Xinwei Fu Gaurav Gupta Dhananjay Ram Yida Wang Jun Huan 81 6 0 16 Apr 2024
I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey Noah Lewis J. L. Bez Suren Byna 109 0 0 16 Apr 2024
AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees William Fleshman Aleem Khan Marc Marone Benjamin Van Durme CLL KELM 122 4 0 12 Apr 2024
Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Languages Vandan Mujadia Pruthwik Mishra Arafat Ahsan D. Sharma ELM 75 2 0 03 Apr 2024
Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization Hritik Bansal Ashima Suvarna Gantavya Bhatt Nanyun Peng Kai-Wei Chang Aditya Grover ALM 153 11 0 31 Mar 2024
Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models Hyunbyung Park Sukyung Lee Gyoungjin Gim Yungi Kim Dahyun Kim Chanjun Park VLM 114 0 0 28 Mar 2024
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance Jiasheng Ye Peiju Liu Tianxiang Sun Yunhua Zhou Jun Zhan Xipeng Qiu 139 76 0 25 Mar 2024
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding Zhenyu Zhang Runjin Chen Shiwei Liu Zhewei Yao Olatunji Ruwase Beidi Chen Xiaoxia Wu Zhangyang Wang 95 36 0 05 Mar 2024
Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs Aly M. Kassem Omar Mahmoud Niloofar Mireshghallah Hyunwoo J. Kim Yulia Tsvetkov Yejin Choi Sherif Saad Santu Rana 129 22 0 05 Mar 2024
Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs Raghavv Goel Mukul Gagrani Wonseok Jeon Junyoung Park Mingu Lee Christopher Lott ALM 63 7 0 29 Feb 2024
ChatMusician: Understanding and Generating Music Intrinsically with LLM Ti-Fen Pan Hanfeng Lin Yi Wang Zeyue Tian Shangda Wu ... Gus Xia Roger Dannenberg Wei Xue Shiyin Kang Yike Guo 174 44 0 25 Feb 2024
LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey Ashok Urlana Charaka Vinayak Kumar Ajeet Kumar Singh B. Garlapati S. Chalamala Rahul Mishra 119 8 0 22 Feb 2024
PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods Slawomir Dadas Michal Perelkiewicz Rafal Poswiata 109 3 0 20 Feb 2024
High-quality Data-to-Text Generation for Severely Under-Resourced Languages with Out-of-the-box Large Language Models Michela Lorandi Anya Belz 53 5 0 19 Feb 2024
Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction Kuniaki Saito Kihyuk Sohn Chen-Yu Lee Yoshitaka Ushiku 143 3 0 16 Feb 2024
Large Language Models: A Survey Shervin Minaee Tomas Mikolov Narjes Nikzad M. Asgari-Chenaghlu R. Socher Xavier Amatriain Jianfeng Gao ALM LM&MA ELM 246 425 0 09 Feb 2024
CroissantLLM: A Truly Bilingual French-English Language Model Manuel Faysse Patrick Fernandes Nuno M. Guerreiro António Loison Duarte M. Alves ... François Yvon André F.T. Martins Gautier Viaud C´eline Hudelot Pierre Colombo 159 37 0 01 Feb 2024
When Large Language Models Meet Vector Databases: A Survey Zhi Jing Yongye Su Yikun Han Bo Yuan Haiyun Xu Chunjiang Liu Kehai Chen Min Zhang 138 38 0 30 Jan 2024
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model Xiao-wen Dong Pan Zhang Yuhang Zang Yuhang Cao Bin Wang ... Conghui He Xingcheng Zhang Yu Qiao Dahua Lin Jiaqi Wang VLM MLLM 156 268 0 29 Jan 2024
MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning Chenyu Wang Weixin Luo Qianyu Chen Haonan Mai Jindi Guo Sixun Dong Xiaohua Xuan MLLM LLMAG 145 20 0 19 Jan 2024
EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysis Zhiwei Liu Kailai Yang Tianlin Zhang Qianqian Xie Sophia Ananiadou 74 52 0 16 Jan 2024
INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning Yutao Zhu Peitian Zhang Chenghao Zhang Yifei Chen Binyu Xie Zheng Liu Ji-Rong Wen Zhicheng Dou 60 17 0 12 Jan 2024
MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition Zheng Lian Guoying Zhao Yong Ren Hao Gu Haiyang Sun Lan Chen Bin Liu Jianhua Tao 124 13 0 07 Jan 2024
Quokka: An Open-source Large Language Model ChatBot for Material Science Xianjun Yang Stephen D. Wilson Linda R. Petzold OSLM 71 2 0 02 Jan 2024
ALMANACS: A Simulatability Benchmark for Language Model Explainability Edmund Mills Shiye Su Stuart J. Russell Scott Emmons 160 9 0 20 Dec 2023
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model Shraman Pramanick Guangxing Han Rui Hou Sayan Nag Ser-Nam Lim Nicolas Ballas Qifan Wang Rama Chellappa Amjad Almahairi VLM MLLM 164 36 0 19 Dec 2023
LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systems Artem Lykov Maria Dronova Nikolay Naglov Mikhail Litvinov Sergei Satsevich Artem Bazhenov Vladimir Berman Aleksei Shcherbak Dzmitry Tsetserukou LLMAG LM&Ro 104 15 0 14 Dec 2023
LLM360: Towards Fully Transparent Open-Source LLMs Zhengzhong Liu Aurick Qiao Willie Neiswanger Hongyi Wang Bowen Tan ... Zhiting Hu Mark Schulze Preslav Nakov Timothy Baldwin Eric Xing 124 78 0 11 Dec 2023
Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models Andrea Papaluca Daniel Krefl Sergio Mendez Rodriguez Artem Lenskiy Hanna Suominen 55 3 0 04 Dec 2023
The Claire French Dialogue Dataset Julie Hunter Jérôme Louradour Virgile Rennard Ismail Harrando Guokan Shang Jean-Pierre Lorré 92 1 0 28 Nov 2023
PrivateLoRA For Efficient Privacy Preserving LLM Yiming Wang Yu Lin Xiaodong Zeng Guannan Zhang 105 14 0 23 Nov 2023
P^3SUM: Preserving Author's Perspective in News Summarization with Diffusion Language Models Yuhan Liu Shangbin Feng Xiaochuang Han Vidhisha Balachandran Chan Young Park Sachin Kumar Yulia Tsvetkov DiffM 86 4 0 16 Nov 2023
R-Tuning: Instructing Large Language Models to Say `I Don't Know' Hanning Zhang Shizhe Diao Yong Lin Yi R. Fung Qing Lian Xingyao Wang Yangyi Chen Heng Ji Tong Zhang UQLM 131 47 0 16 Nov 2023
SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models Bertie Vidgen Nino Scherrer Hannah Rose Kirk Rebecca Qian Anand Kannappan Scott A. Hale Paul Röttger ALM ELM 116 29 0 14 Nov 2023
Intentional Biases in LLM Responses Nicklaus Badyal Derek Jacoby Yvonne Coady 55 5 0 11 Nov 2023