v1v2v3v4 (latest)

Compute-Constrained Data Selection

21 October 2024

Papers citing "Compute-Constrained Data Selection"

41 / 41 papers shown

Title
Efficient Data Selection at Scale via Influence Distillation Mahdi Nikdan Vincent Cohen-Addad Dan Alistarh Vahab Mirrokni TDI 53 0 0 25 May 2025
Scaling Laws for Downstream Task Performance in Machine Translation Berivan Isik Natalia Ponomareva Hussein Hazimeh Dimitris Paparas Sergei Vassilvitskii Sanmi Koyejo 158 23 0 24 Feb 2025
Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models Zachary Ankner Cody Blakeney Kartik K. Sreenivasan Max Marion Matthew L. Leavitt Mansheej Paul 98 34 0 30 May 2024
Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic Sachin Goyal Pratyush Maini Zachary Chase Lipton Aditi Raghunathan J. Zico Kolter 95 46 0 10 Apr 2024
When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method Biao Zhang Zhongtao Liu Colin Cherry Orhan Firat LRM 104 153 0 27 Feb 2024
QuRating: Selecting High-Quality Data for Training Language Models Alexander Wettig Aatmik Gupta Saumya Malik Danqi Chen 52 77 0 15 Feb 2024
LESS: Selecting Influential Data for Targeted Instruction Tuning Mengzhou Xia Sadhika Malladi Suchin Gururangan Sanjeev Arora Danqi Chen 153 243 0 06 Feb 2024
Selecting Large Language Model to Fine-tune via Rectified Scaling Law Haowei Lin Baizhou Huang Haotian Ye Qinyu Chen Zihao Wang Sujian Li Jianzhu Ma Xiaojun Wan James Zou Yitao Liang 123 20 0 04 Feb 2024
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism DeepSeek-AI Xiao Bi : Xiao Bi Deli Chen Guanting Chen ... Yao Zhao Shangyan Zhou Shunfeng Zhou Qihao Zhu Yuheng Zou LRM ALM 197 377 0 05 Jan 2024
Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2 Hamish Ivison Yizhong Wang Valentina Pyatkin Nathan Lambert Matthew E. Peters ... Joel Jang David Wadden Noah A. Smith Iz Beltagy Hanna Hajishirzi ALM ELM 96 194 0 17 Nov 2023
Instruction-Following Evaluation for Large Language Models Jeffrey Zhou Tianjian Lu Swaroop Mishra Siddhartha Brahma Sujoy Basu Yi Luan Denny Zhou Le Hou ELM ALM LRM 84 296 0 14 Nov 2023
#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models Keming Lu Hongyi Yuan Zheng Yuan Runji Lin Junyang Lin Chuanqi Tan Chang Zhou Jingren Zhou ALM LRM 53 75 0 14 Aug 2023
In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuning Xiaochuang Han 42 19 0 08 Aug 2023
Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models Mayee F. Chen Nicholas Roberts Kush S. Bhatia Jue Wang Ce Zhang Frederic Sala Christopher Ré SyDa 73 65 0 26 Jul 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 399 12,044 0 18 Jul 2023
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources Yizhong Wang Hamish Ivison Pradeep Dasigi Jack Hessel Tushar Khot ... David Wadden Kelsey MacMillan Noah A. Smith Iz Beltagy Hannaneh Hajishirzi ALM ELM 113 393 0 07 Jun 2023
Scaling Data-Constrained Language Models Niklas Muennighoff Alexander M. Rush Boaz Barak Teven Le Scao Aleksandra Piktus Nouamane Tazi S. Pyysalo Thomas Wolf Colin Raffel ALM 106 224 0 25 May 2023
CodeT5+: Open Code Large Language Models for Code Understanding and Generation Yue Wang Hung Le Akhilesh Deepak Gotmare Nghi D. Q. Bui Junnan Li Steven C. H. Hoi ALM 82 491 0 13 May 2023
OpenAssistant Conversations -- Democratizing Large Language Model Alignment Andreas Kopf Yannic Kilcher Dimitri von Rutte Sotiris Anagnostidis Zhi Rui Tam ... Arnav Dantuluri Andrew Maguire Christoph Schuhmann Huu Nguyen A. Mattick ALM LM&MA 144 638 0 14 Apr 2023
Data Selection for Language Models via Importance Resampling Sang Michael Xie Shibani Santurkar Tengyu Ma Percy Liang 118 195 0 06 Feb 2023
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning Shayne Longpre Le Hou Tu Vu Albert Webson Hyung Won Chung ... Denny Zhou Quoc V. Le Barret Zoph Jason W. Wei Adam Roberts ALM 112 677 0 31 Jan 2023
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors Hamish Ivison Noah A. Smith Hannaneh Hajishirzi Pradeep Dasigi 92 22 0 01 Dec 2022
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 218 3,158 0 20 Oct 2022
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them Mirac Suzgun Nathan Scales Nathanael Scharli Sebastian Gehrmann Yi Tay ... Aakanksha Chowdhery Quoc V. Le Ed H. Chi Denny Zhou Jason W. Wei ALM ELM LRM ReLM 268 1,139 0 17 Oct 2022
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models Aarohi Srivastava Abhinav Rastogi Abhishek Rao Abu Awal Md Shoeb Abubakar Abid ... Zhuoye Zhao Zijian Wang Zijie J. Wang Zirui Wang Ziyi Wu ELM 211 1,775 0 09 Jun 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 208 1,980 0 29 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 845 9,683 0 28 Jan 2022
Learning To Retrieve Prompts for In-Context Learning Ohad Rubin Jonathan Herzig Jonathan Berant VPVLM RALM 86 709 0 16 Dec 2021
Large Dual Encoders Are Generalizable Retrievers Jianmo Ni Chen Qu Jing Lu Zhuyun Dai Gustavo Hernández Ábrego ... Vincent Zhao Yi Luan Keith B. Hall Ming-Wei Chang Yinfei Yang DML 167 461 0 15 Dec 2021
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 355 1,709 0 15 Oct 2021
Finetuned Language Models Are Zero-Shot Learners Jason W. Wei Maarten Bosma Vincent Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai Quoc V. Le ALM UQCV 233 3,782 0 03 Sep 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 490 10,496 0 17 Jun 2021
Cross-Task Generalization via Natural Language Crowdsourcing Instructions Swaroop Mishra Daniel Khashabi Chitta Baral Hannaneh Hajishirzi LRM 171 752 0 18 Apr 2021
GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training Krishnateja Killamsetty D. Sivasubramanian Ganesh Ramakrishnan A. De Rishabh K. Iyer OOD 147 206 0 27 Feb 2021
Scaling Laws for Transfer Danny Hernandez Jared Kaplan T. Henighan Sam McCandlish 92 250 0 02 Feb 2021
What Makes Good In-Context Examples for GPT- $3$ ? Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin Weizhu Chen AAML RALM 390 1,390 0 17 Jan 2021
GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning Krishnateja Killamsetty D. Sivasubramanian Ganesh Ramakrishnan Rishabh Iyer University of Texas at Dallas 74 211 0 19 Dec 2020
Measuring Massive Multitask Language Understanding Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Basel Alomair Jacob Steinhardt ELM RALM 184 4,572 0 07 Sep 2020
Selecting Informative Contexts Improves Language Model Finetuning Richard Antonello Nicole M. Beckage Javier S. Turek Alexander G. Huth 79 11 0 01 May 2020
Estimating Training Data Influence by Tracing Gradient Descent G. Pruthi Frederick Liu Mukund Sundararajan Satyen Kale TDI 99 417 0 19 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 611 4,921 0 23 Jan 2020