Title
DataRater: Meta-Learned Dataset Curation Dan A. Calian Gregory Farquhar Iurii Kemaev Luisa M. Zintgraf Matteo Hessel ... András Gyorgy Tom Schaul Jeffrey Dean Hado van Hasselt David Silver 117 0 0 23 May 2025
Understanding Gated Neurons in Transformers from Their Input-Output Functionality Sebastian Gerstner Hinrich Schütze MILM FAtt 117 0 0 23 May 2025
Diagnosing our datasets: How does my language model learn clinical information? Furong Jia David Sontag Monica Agrawal LM&MA 123 0 0 21 May 2025
Safety Pretraining: Toward the Next Generation of Safe AI Pratyush Maini Sachin Goyal Dylan Sam Alex Robey Yash Savani Yiding Jiang Andy Zou Zacharcy C. Lipton J. Zico Kolter 112 0 0 23 Apr 2025
Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models Xinlin Zhuang Jiahui Peng Ren Ma Yucheng Wang Tianyi Bai Xingjian Wei Jiantao Qiu Chi Zhang Ying Qian Conghui He 86 0 0 19 Apr 2025
SEA-LION: Southeast Asian Languages in One Network Raymond Ng Thanh Ngan Nguyen Yuli Huang Ngee Chia Tai Wai Yi Leong ... David Ong Tat-Wee B. Liu William-Chandra Tjhi Min Zhang Leslie Teo 62 13 0 08 Apr 2025
Not All Data Are Unlearned Equally Aravind Krishnan Siva Reddy Marius Mosbach MU 290 1 0 07 Apr 2025
ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selection Xiaoxuan Zhu Zhouhong Gu Baiqian Wu Suhang Zheng Tao Wang Tianyu Li Hongwei Feng Yanghua Xiao 124 0 0 01 Apr 2025
Predictive Data Selection: The Data That Predicts Is the Data That Teaches Kashun Shum Yuanmin Huang Hongjian Zou Qi Ding Yixuan Liao Xiao Chen Qian Liu Junxian He 100 3 0 02 Mar 2025
TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking Shahriar Kabir Nahin R. N. Nandi Sagor Sarker Quazi Sarwar Muhtaseem Md. Kowsher Apu Chandraw Shill Md Ibrahim Mehadi Hasan Menon Tareq Al Muntasir Firoj Alam 117 0 0 24 Feb 2025
TinyEmo: Scaling down Emotional Reasoning via Metric Projection Cristian Gutierrez LRM 140 0 0 17 Feb 2025
Do we really have to filter out random noise in pre-training data for language models? Jinghan Ru Yuxin Xie Xianwei Zhuang Yuguo Yin Zhihui Guo Zhiming Liu Qianli Ren Yuexian Zou 133 4 0 10 Feb 2025
Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related tasks Jing Yang Max Glockner Anderson de Rezende Rocha Iryna Gurevych LRM 95 1 0 07 Feb 2025
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise Rose E. Wang Ana T. Ribeiro Carly Robinson Susanna Loeb Dora Demszky 112 14 0 28 Jan 2025
NExtLong: Toward Effective Long-Context Training without Long Documents Chaochen Gao Xing Wu Zijia Lin Debing Zhang Songlin Hu SyDa 109 2 0 22 Jan 2025
Merging Feed-Forward Sublayers for Compressed Transformers Neha Verma Kenton W. Murray Kevin Duh AI4CE 89 0 0 10 Jan 2025
Social Science Is Necessary for Operationalizing Socially Responsible Foundation Models Adam Davies Elisa Nguyen Michael Simeone Erik Johnston Martin Gubri 144 0 0 20 Dec 2024
DateLogicQA: Benchmarking Temporal Biases in Large Language Models Gagan Bhatia MingZe Tang Cristina Mahanta Madiha Kazi 101 0 0 17 Dec 2024
LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation Eunsu Kim Juyoung Suk Seungone Kim Niklas Muennighoff Dongkwan Kim Alice Oh ELM 115 1 0 10 Dec 2024
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Akhiad Bercovich Tomer Ronen Talor Abramovich Nir Ailon Nave Assaf ... Ido Shahaf Oren Tropp Omer Ullman Argov Ran Zilberstein Ran El-Yaniv 141 3 0 28 Nov 2024
LLäMmlein: Compact and Competitive German-Only Language Models from Scratch Jan Pfister Julia Wunderle Andreas Hotho 60 2 0 17 Nov 2024
LSHBloom: Memory-efficient, Extreme-scale Document Deduplication A. Khan Robert Underwood Carlo Siebenschuh Y. Babuji Aswathy Ajith Kyle Hippe Ozan Gokdemir Alexander Brace Kyle Chard Ian Foster 52 0 0 06 Nov 2024
Sparsing Law: Towards Large Language Models with Greater Activation Sparsity Yuqi Luo Chenyang Song Xu Han Yuxiao Chen Chaojun Xiao Zhiyuan Liu Maosong Sun 75 4 0 04 Nov 2024
GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages Amir Hossein Kargaran François Yvon Hinrich Schutze VLM 72 7 0 31 Oct 2024
COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training Haocheng Xi Han Cai Ligeng Zhu Yaojie Lu Kurt Keutzer Jianfei Chen Song Han MQ 93 9 0 25 Oct 2024
Cross-lingual Transfer of Reward Models in Multilingual Alignment Jiwoo Hong Noah Lee Rodrigo Martínez-Castaño César Rodríguez James Thorne 73 5 0 23 Oct 2024
Detecting Training Data of Large Language Models via Expectation Maximization Gyuwan Kim Yang Li Evangelia Spiliopoulou Jie Ma Miguel Ballesteros William Yang Wang MIALM 150 4 2 10 Oct 2024
Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models Zhipeng Chen Liang Song K. Zhou Wayne Xin Zhao Binghai Wang Weipeng Chen Ji-Rong Wen 81 0 0 10 Oct 2024
How Much Can We Forget about Data Contamination? Sebastian Bordt Suraj Srinivas Valentyn Boreiko U. V. Luxburg 85 2 0 04 Oct 2024
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 114 45 0 03 Oct 2024
Training Language Models on Synthetic Edit Sequences Improves Code Synthesis Ulyana Piterbarg Lerrel Pinto Rob Fergus SyDa 51 2 0 03 Oct 2024
Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition Jiyeon Kim Hyunji Lee Hyowon Cho Joel Jang Hyeonbin Hwang Seungpil Won Youbin Ahn Dohaeng Lee Minjoon Seo KELM 316 4 0 02 Oct 2024
Small Language Models: Survey, Measurements, and Insights Zhenyan Lu Xiang Li Dongqi Cai Rongjie Yi Fangming Liu Xiwen Zhang Nicholas D. Lane Mengwei Xu ObjD LRM 89 44 0 24 Sep 2024
Improving Pretraining Data Using Perplexity Correlations Tristan Thrush Christopher Potts Tatsunori Hashimoto 68 19 0 09 Sep 2024
How Does Code Pretraining Affect Language Model Task Performance? Jackson Petty Sjoerd van Steenkiste Tal Linzen 75 10 0 06 Sep 2024
An Investigation of Warning Erroneous Chat Translations in Cross-lingual Communication Yunmeng Li Jun Suzuki Makoto Morishita Kaori Abe Kentaro Inui 92 1 0 28 Aug 2024
RegMix: Data Mixture as Regression for Language Model Pre-training Qian Liu Xiaosen Zheng Niklas Muennighoff Guangtao Zeng Longxu Dou Tianyu Pang Jing Jiang Min Lin MoE 101 49 1 01 Jul 2024
Blind Baselines Beat Membership Inference Attacks for Foundation Models Debeshee Das Jie Zhang Florian Tramèr MIALM 116 32 1 23 Jun 2024
Emergence of a High-Dimensional Abstraction Phase in Language Transformers Emily Cheng Diego Doimo Corentin Kervadec Iuri Macocco Jade Yu Alessandro Laio Marco Baroni 117 13 0 24 May 2024
DEPTH: Discourse Education through Pre-Training Hierarchically Zachary Bamberger Ofek Glick Chaim Baskin Yonatan Belinkov 94 0 0 13 May 2024
Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization Hritik Bansal Ashima Suvarna Gantavya Bhatt Nanyun Peng Kai-Wei Chang Aditya Grover ALM 82 10 0 31 Mar 2024
Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models Hyunbyung Park Sukyung Lee Gyoungjin Gim Yungi Kim Dahyun Kim Chanjun Park VLM 62 0 0 28 Mar 2024
Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs Aly M. Kassem Omar Mahmoud Niloofar Mireshghallah Hyunwoo J. Kim Yulia Tsvetkov Yejin Choi Sherif Saad Santu Rana 77 20 0 05 Mar 2024
Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens Jiacheng Liu Sewon Min Luke Zettlemoyer Yejin Choi Hannaneh Hajishirzi 86 54 0 30 Jan 2024