Title
L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models Xiaohao Liu Xiaobo Xia Weixiang Zhao Manyi Zhang Xianzhi Yu Xiu Su Shuo Yang See-Kiong Ng Tat-Seng Chua KELM LRM 69 0 0 23 May 2025
ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs Fahmida Liza Piya Rahmatollah Beheshti 202 0 0 23 Apr 2025
Looking beyond the next token Abitha Thankaraj Yiding Jiang J. Zico Kolter Yonatan Bisk LRM 76 1 0 15 Apr 2025
Information-Theoretic Reward Decomposition for Generalizable RLHF Liyuan Mao Haoran Xu Amy Zhang Weinan Zhang Chenjia Bai 78 0 0 08 Apr 2025
Scaling Laws for Downstream Task Performance in Machine Translation Berivan Isik Natalia Ponomareva Hussein Hazimeh Dimitris Paparas Sergei Vassilvitskii Sanmi Koyejo 137 23 0 24 Feb 2025
ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval Guanqi Zhan Yuanpei Liu Kai Han Weidi Xie Andrew Zisserman VLM 413 0 0 21 Feb 2025
Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale Fan Zhou Zengzhi Wang Qian Liu Junlong Li Pengfei Liu ALM 146 14 0 17 Feb 2025
One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs Hai-Tao Zheng Jiayi Kuang Haojing Huang Zhikun Xu Xinnian Liang ... Jue Chen Chao Qu Ying Shen Hai-Tao Zheng Philip S. Yu LRM 88 2 0 12 Feb 2025
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions Moran Yanuka Assaf Ben-Kish Yonatan Bitton Idan Szpektor Raja Giryes VLM 75 3 0 13 Nov 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 95 12 0 31 Oct 2024
High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws M. E. Ildiz Halil Alperen Gozeten Ege Onur Taga Marco Mondelli Samet Oymak 79 5 0 24 Oct 2024
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models Michael Noukhovitch Shengyi Huang Sophie Xhonneux Arian Hosseini Rishabh Agarwal Rameswar Panda OffRL 109 10 0 23 Oct 2024
ToW: Thoughts of Words Improve Reasoning in Large Language Models Zhikun Xu Ming shen Jacob Dineen Zhaonan Li Xiao Ye Shijie Lu Aswin Rrv Chitta Baral Ben Zhou LRM 379 1 0 21 Oct 2024
Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning Jiacheng Ye Jiahui Gao Shansan Gong Lin Zheng Xin Jiang Zhiyu Li Dianbo Sui DiffM LRM 117 21 0 18 Oct 2024
CursorCore: Assist Programming through Aligning Anything Hao Jiang Qi Liu Rui Li Shengyu Ye Shijin Wang 77 1 0 09 Oct 2024
Data Selection via Optimal Control for Language Models Yuxian Gu Li Dong Hongning Wang Y. Hao Qingxiu Dong Furu Wei Minlie Huang AI4CE 87 6 0 09 Oct 2024
SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe Yuxin Xiao Shujian Zhang Wenxuan Zhou Marzyeh Ghassemi Sanqiang Zhao 327 0 0 07 Oct 2024
Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning Yeongbin Seo Dongha Lee Jinyoung Yeo CLL KELM 127 2 0 24 Jul 2024
RegMix: Data Mixture as Regression for Language Model Pre-training Qian Liu Xiaosen Zheng Niklas Muennighoff Guangtao Zeng Longxu Dou Tianyu Pang Jing Jiang Min Lin MoE 111 49 1 01 Jul 2024
Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models Sander Land Max Bartolo 63 21 0 08 May 2024
Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs Feiyang Kang H. Just Yifan Sun Himanshu Jahagirdar Yuanzhi Zhang Rongxing Du Anit Kumar Sahu Ruoxi Jia 61 21 0 05 May 2024
Exploring the Mystery of Influential Data for Mathematical Reasoning Xinzhe Ni Yeyun Gong Zhibin Gou Yelong Shen Yujiu Yang Nan Duan Weizhu Chen 59 10 0 01 Apr 2024
Language models scale reliably with over-training and on downstream tasks S. Gadre Georgios Smyrnis Vaishaal Shankar Suchin Gururangan Mitchell Wortsman ... Y. Carmon Achal Dave Reinhard Heckel Niklas Muennighoff Ludwig Schmidt ALM ELM LRM 121 44 0 13 Mar 2024
Gemma: Open Models Based on Gemini Research and Technology Gemma Team Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju ... Armand Joulin Noah Fiedel Evan Senter Alek Andreev Kathleen Kenealy VLM LLMAG 168 473 0 13 Mar 2024
Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning Yiming Huang Xiao Liu Yeyun Gong Zhibin Gou Yelong Shen Nan Duan Weizhu Chen AIMat LRM 61 38 0 04 Mar 2024
InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning Huaiyuan Ying Shuo Zhang Linyang Li Zhejian Zhou Yunfan Shao ... Hang Yan Xipeng Qiu Jiayu Wang Kai-xiang Chen Dahua Lin ReLM LRM 47 81 0 09 Feb 2024
LESS: Selecting Influential Data for Targeted Instruction Tuning Mengzhou Xia Sadhika Malladi Suchin Gururangan Sanjeev Arora Danqi Chen 121 230 0 06 Feb 2024
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models Zhihong Shao Peiyi Wang Qihao Zhu Runxin Xu Jun-Mei Song ... Haowei Zhang Mingchuan Zhang Yiming Li Yu-Huan Wu Daya Guo ReLM LRM 96 1,015 0 05 Feb 2024
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong ... Yu-Huan Wu Yiming Li Fuli Luo Yingfei Xiong W. Liang ELM 98 748 0 25 Jan 2024
DsDm: Model-Aware Dataset Selection with Datamodels Logan Engstrom Axel Feldmann Aleksander Madry OODD 56 55 0 23 Jan 2024
Knowledge Fusion of Large Language Models Fanqi Wan Xinting Huang Deng Cai Xiaojun Quan Wei Bi Shuming Shi MoMe 79 69 0 19 Jan 2024
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism DeepSeek-AI Xiao Bi : Xiao Bi Deli Chen Guanting Chen ... Yao Zhao Shangyan Zhou Shunfeng Zhou Qihao Zhu Yuheng Zou LRM ALM 174 358 0 05 Jan 2024
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning Wei Liu Weihao Zeng Keqing He Yong Jiang Junxian He ALM 80 231 0 25 Dec 2023
One-Shot Learning as Instruction Data Prospector for Large Language Models Yunshui Li Binyuan Hui Xiaobo Xia Jiaxi Yang Min Yang ... Ling-Hao Chen Junhao Liu Tongliang Liu Fei Huang Yongbin Li 58 36 0 16 Dec 2023
Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2 Hamish Ivison Yizhong Wang Valentina Pyatkin Nathan Lambert Matthew E. Peters ... Joel Jang David Wadden Noah A. Smith Iz Beltagy Hanna Hajishirzi ALM ELM 73 192 0 17 Nov 2023
Irreducible Curriculum for Language Model Pretraining Simin Fan Martin Jaggi 112 10 0 23 Oct 2023
Llemma: An Open Language Model For Mathematics Zhangir Azerbayev Hailey Schoelkopf Keiran Paster Marco Dos Santos Stephen Marcus McAleer Albert Q. Jiang Jia Deng Stella Biderman Sean Welleck CLL 68 294 0 16 Oct 2023
Mistral 7B Albert Q. Jiang Alexandre Sablayrolles A. Mensch Chris Bamford Devendra Singh Chaplot ... Teven Le Scao Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LRM 51 2,151 0 10 Oct 2023
Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models Tianjian Li Haoran Xu Philipp Koehn Daniel Khashabi Kenton W. Murray 53 4 0 02 Oct 2023
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving Zhibin Gou Zhihong Shao Yeyun Gong Yelong Shen Yujiu Yang Minlie Huang Nan Duan Weizhu Chen LRM AI4CE LLMAG 75 156 0 29 Sep 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 184 1,781 0 28 Sep 2023
At Which Training Stage Does Code Data Help LLMs Reasoning? Xiaogang Jia Yue Liu Yue Yu Yuanliang Zhang Yu Jiang Changjian Wang Shanshan Li LRM SyDa 52 65 0 28 Sep 2023
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models L. Yu Weisen Jiang Han Shi Jincheng Yu Zhengying Liu Yu Zhang James T. Kwok Zheng Li Adrian Weller Weiyang Liu OSLM LRM 82 376 0 21 Sep 2023
Efficient Memory Management for Large Language Model Serving with PagedAttention Woosuk Kwon Zhuohan Li Siyuan Zhuang Ying Sheng Lianmin Zheng Cody Hao Yu Joseph E. Gonzalez Haotong Zhang Ion Stoica VLM 147 2,162 0 12 Sep 2023
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning Xiang Yue Xingwei Qu Ge Zhang Yao Fu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen AIMat LRM 115 395 0 11 Sep 2023
Code Llama: Open Foundation Models for Code Baptiste Rozière Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat ... Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom Gabriel Synnaeve ELM ALM 91 2,011 0 24 Aug 2023
D4: Improving LLM Pretraining via Document De-Duplication and Diversification Kushal Tirumala Daniel Simig Armen Aghajanyan Ari S. Morcos SyDa 41 111 0 23 Aug 2023
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning Ming Li Yong Zhang Zhitao Li Jiuhai Chen Lichang Chen Ning Cheng Jianzong Wang Dinesh Manocha Jing Xiao 102 202 0 23 Aug 2023
Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models Mayee F. Chen Nicholas Roberts Kush S. Bhatia Jue Wang Ce Zhang Frederic Sala Christopher Ré SyDa 51 59 0 26 Jul 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 66 42 0 12 Jul 2023