Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale

17 February 2025

Papers citing "Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale"

50 / 75 papers shown

Title
Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data Yiwen Liu Jessica Bader Jae Myung Kim DiffM 70 1 0 15 May 2025
MegaMath: Pushing the Limits of Open Math Corpora Fan Zhou Zengzhi Wang Nikhil Ranjan Zhoujun Cheng Liping Tang Guowei He Zhengzhong Liu Eric P. Xing LRM 116 3 0 03 Apr 2025
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling Haebin Shin Lei Ji Xiao Liu Yeyun Gong 103 0 0 24 Mar 2025
MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models Jia-Nan Li Lu Yu Daixin Wang Qing Cui Jun Zhou Yanfang Ye Chuxu Zhang 114 0 0 19 Mar 2025
SkyLadder: Better and Faster Pretraining via Context Window Scheduling Tongyao Zhu Qian Liu Haonan Wang Shiqi Chen Xiangming Gu Tianyu Pang Min-Yen Kan 84 0 0 19 Mar 2025
Predictive Data Selection: The Data That Predicts Is the Data That Teaches Kashun Shum Yuanmin Huang Hongjian Zou Qi Ding Yixuan Liao Xiao Chen Qian Liu Junxian He 163 4 0 02 Mar 2025
Evolving Symbolic 3D Visual Grounder with Weakly Supervised Reflection Boyu Mi Hanqing Wang Tai Wang Yilun Chen Jiangmiao Pang 112 0 0 21 Feb 2025
Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs Longxu Dou Qian Liu Fan Zhou Changyu Chen Zili Wang ... Tianyu Pang Chao Du Xinyi Wan Wei Lu Min Lin 217 3 0 18 Feb 2025
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling Bradley Brown Jordan Juravsky Ryan Ehrlich Ronald Clark Quoc V. Le Christopher Ré Azalia Mirhoseini ALM LRM 249 330 0 03 Jan 2025
Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions Yutao Hou Yajing Luo Zhiwen Ruan Hongru Wang Weifeng Ge Yuxiao Chen Guanhua Chen ELM 76 0 0 15 Nov 2024
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models Siming Huang Tianhao Cheng J.K. Liu Jiaran Hao L. Song ... Ge Zhang Zili Wang Yuan Qi Yinghui Xu Wei Chu ALM 152 31 0 07 Nov 2024
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Charlie Snell Jaehoon Lee Kelvin Xu Aviral Kumar LRM 198 698 0 06 Aug 2024
RegMix: Data Mixture as Regression for Language Model Pre-training Qian Liu Xiaosen Zheng Niklas Muennighoff Guangtao Zeng Longxu Dou Tianyu Pang Jing Jiang Min Lin MoE 163 54 1 01 Jul 2024
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale Guilherme Penedo Hynek Kydlícek Loubna Ben Allal Anton Lozhkov Margaret Mitchell Colin Raffel Leandro von Werra Thomas Wolf 121 265 0 25 Jun 2024
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions Terry Yue Zhuo Minh Chien Vu Jenny Chim Han Hu Wenhao Yu ... David Lo Daniel Fried Xiaoning Du H. D. Vries Leandro von Werra 172 192 0 22 Jun 2024
Instruction Pre-Training: Language Models are Supervised Multitask Learners Daixuan Cheng Yuxian Gu Shaohan Huang Junyu Bi Minlie Huang Furu Wei SyDa 123 24 0 20 Jun 2024
MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models Zichun Yu Spandan Das Chenyan Xiong 107 36 0 10 Jun 2024
Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models Zachary Ankner Cody Blakeney Kartik K. Sreenivasan Max Marion Matthew L. Leavitt Mansheej Paul 98 34 0 30 May 2024
MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series Ge Zhang Scott Qu Jiaheng Liu Chenchen Zhang Chenghua Lin ... Zi-Kai Zhao Jiajun Zhang Wanli Ouyang Wenhao Huang Wenhu Chen ELM 97 46 0 29 May 2024
MAmmoTH2: Scaling Instructions from the Web Xiang Yue Tuney Zheng Ge Zhang Wenhu Chen ALM LRM 94 101 0 06 May 2024
Rho-1: Not All Tokens Are What You Need Zheng-Wen Lin Zhibin Gou Yeyun Gong Xiao Liu Yelong Shen ... Chen Lin Yujiu Yang Jian Jiao Nan Duan Weizhu Chen CLL 123 75 0 11 Apr 2024
Best Practices and Lessons Learned on Synthetic Data for Language Models Ruibo Liu Jerry W. Wei Fangyu Liu Chenglei Si Yanzhe Zhang ... Steven Zheng Daiyi Peng Diyi Yang Denny Zhou Andrew M. Dai SyDa EgoV 119 95 0 11 Apr 2024
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies Shengding Hu Yuge Tu Xu Han Chaoqun He Ganqu Cui ... Chaochao Jia Guoyang Zeng Dahai Li Zhiyuan Liu Maosong Sun MoE 105 345 0 09 Apr 2024
Sailor: Open Language Models for South-East Asia Longxu Dou Qian Liu Guangtao Zeng Jia Guo Jiahui Zhou Wei Lu Min Lin LRM 96 9 0 04 Apr 2024
The Larger the Better? Improved LLM Code-Generation via Budget Reallocation Michael Hassid Tal Remez Jonas Gehring Roy Schwartz Yossi Adi 97 27 0 31 Mar 2024
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models Yaowei Zheng Richong Zhang Junhao Zhang Yanhan Ye Zheyan Luo Zhangchi Feng Yongqiang Ma 153 553 0 20 Mar 2024
WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset Jiantao Qiu Haijun Lv Zhenjiang Jin Rui Wang Wenchang Ning ... Zhongying Tu Lin Dahua Yu Qiao Hang Yan Conghui He 68 7 0 29 Feb 2024
Reformatted Alignment Run-Ze Fan Xuefeng Li Haoyang Zou Junlong Li Shwai He Ethan Chern Jiewen Hu Pengfei Liu 92 7 0 19 Feb 2024
QuRating: Selecting High-Quality Data for Training Language Models Alexander Wettig Aatmik Gupta Saumya Malik Danqi Chen 52 79 0 15 Feb 2024
How to Train Data-Efficient LLMs Noveen Sachdeva Benjamin Coleman Wang-Cheng Kang Jianmo Ni Lichan Hong Ed H. Chi James Caverlee Julian McAuley D. Cheng 85 64 0 15 Feb 2024
InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning Huaiyuan Ying Shuo Zhang Linyang Li Zhejian Zhou Yunfan Shao ... Hang Yan Xipeng Qiu Jiayu Wang Kai-xiang Chen Dahua Lin ReLM LRM 65 85 0 09 Feb 2024
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models Zhihong Shao Peiyi Wang Qihao Zhu Runxin Xu Jun-Mei Song ... Haowei Zhang Mingchuan Zhang Yiming Li Yu-Huan Wu Daya Guo ReLM LRM 157 1,287 0 05 Feb 2024
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling Pratyush Maini Skyler Seto Richard He Bai David Grangier Yizhe Zhang Navdeep Jaitly SyDa 80 66 0 29 Jan 2024
DsDm: Model-Aware Dataset Selection with Datamodels Logan Engstrom Axel Feldmann Aleksander Madry OODD 90 61 0 23 Jan 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 367 338 0 18 Jan 2024
TinyLlama: An Open-Source Small Language Model Peiyuan Zhang Guangtao Zeng Tianduo Wang Wei Lu ALM LRM 147 406 0 04 Jan 2024
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning Wei Liu Weihao Zeng Keqing He Yong Jiang Junxian He ALM 101 239 0 25 Dec 2023
FinGPT: Large Generative Models for a Small Language Risto Luukkonen Ville Komulainen Jouni Luoma Anni Eskelinen Jenna Kanerva ... Mikko Merioksa Jyrki Heinonen Aija Vahtola Samuel Antao S. Pyysalo LM&MA 51 49 0 03 Nov 2023
Llemma: An Open Language Model For Mathematics Zhangir Azerbayev Hailey Schoelkopf Keiran Paster Marco Dos Santos Stephen Marcus McAleer Albert Q. Jiang Jia Deng Stella Biderman Sean Welleck CLL 93 302 0 16 Oct 2023
Mistral 7B Albert Q. Jiang Alexandre Sablayrolles A. Mensch Chris Bamford Devendra Singh Chaplot ... Teven Le Scao Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LRM 104 2,238 0 10 Oct 2023
OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text Keiran Paster Marco Dos Santos Zhangir Azerbayev Jimmy Ba LRM 70 91 0 10 Oct 2023
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning Mengzhou Xia Tianyu Gao Zhiyuan Zeng Danqi Chen 121 310 0 10 Oct 2023
Efficient Memory Management for Large Language Model Serving with PagedAttention Woosuk Kwon Zhuohan Li Siyuan Zhuang Ying Sheng Lianmin Zheng Cody Hao Yu Joseph E. Gonzalez Haotong Zhang Ion Stoica VLM 192 2,322 0 12 Sep 2023
Textbooks Are All You Need II: phi-1.5 technical report Yuan-Fang Li Sébastien Bubeck Ronen Eldan Allison Del Giorno Suriya Gunasekar Yin Tat Lee ALM LRM 161 482 0 11 Sep 2023
Code Llama: Open Foundation Models for Code Baptiste Rozière Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat ... Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom Gabriel Synnaeve ELM ALM 125 2,077 0 24 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 400 12,076 0 18 Jul 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao LRM 117 1,326 0 17 Jul 2023
Textbooks Are All You Need Suriya Gunasekar Yi Zhang J. Aneja C. C. T. Mendes Allison Del Giorno ... Sébastien Bubeck Ronen Eldan Adam Tauman Kalai Y. Lee Yuan-Fang Li AI4CE ALM SyDa 88 410 0 20 Jun 2023
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel Yanli Zhao Andrew Gu R. Varma Liangchen Luo Chien-chin Huang ... Bernard Nguyen Geeta Chauhan Y. Hao Ajit Mathews Shen Li FedML MoE 96 345 0 21 Apr 2023
Generative Agents: Interactive Simulacra of Human Behavior J. Park Joseph C. O'Brien Carrie J. Cai Meredith Ringel Morris Percy Liang Michael S. Bernstein LM&Ro AI4CE 404 1,975 0 07 Apr 2023