v1v2 (latest)

Fewer Truncations Improve Language Modeling

16 April 2024

Papers citing "Fewer Truncations Improve Language Modeling"

30 / 30 papers shown

Title
How Well Can General Vision-Language Models Learn Medicine By Watching Public Educational Videos? Rahul Thapa Andrew Li Qingyang Wu Bryan He Yuki Sahashi ... Angela Zhang Ben Athiwaratkun Shuaiwen Leon Song David Ouyang James Zou LM&MA 153 0 0 19 Apr 2025
360Brew: A Decoder-only Foundation Model for Personalized Ranking and Recommendation Hamed Firooz Maziar Sanjabi Adrian Englhardt Aman Gupta Ben Levine ... Xiaoling Zhai Ya Xu Yu Wang Yun Dai Yun Dai ALM 112 4 0 27 Jan 2025
Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models Zihan Qiu Zeyu Huang Jian Xu Kaiyue Wen Zhaoxiang Wang Rui Men Ivan Titov Dayiheng Liu Jingren Zhou Junyang Lin MoE 110 7 0 21 Jan 2025
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 149 47 0 03 Oct 2024
DEPTH: Discourse Education through Pre-Training Hierarchically Zachary Bamberger Ofek Glick Chaim Baskin Yonatan Belinkov 102 0 0 13 May 2024
Data-Juicer: A One-Stop Data Processing System for Large Language Models Daoyuan Chen Yilun Huang Zhijian Ma Hesen Chen Xuchen Pan ... Zhaoyang Liu Jinyang Gao Yaliang Li Bolin Ding Jingren Zhou SyDa VLM 65 37 0 05 Sep 2023
Large Language Models Struggle to Learn Long-Tail Knowledge Nikhil Kandpal H. Deng Adam Roberts Eric Wallace Colin Raffel RALM KELM 126 415 0 15 Nov 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 489 6,240 0 05 Apr 2022
Ethical and social risks of harm from Language Models Laura Weidinger John F. J. Mellor Maribeth Rauh Conor Griffin J. Uesato ... Lisa Anne Hendricks William S. Isaac Sean Legassick G. Irving Iason Gabriel PILM 111 1,036 0 08 Dec 2021
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization Philippe Laban Tobias Schnabel Paul N. Bennett Marti A. Hearst HILM 104 395 0 18 Nov 2021
Entity-Based Knowledge Conflicts in Question Answering Shayne Longpre Kartik Perisetla Anthony Chen Nikhil Ramesh Chris DuBois Sameer Singh HILM 326 260 0 10 Sep 2021
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 195 1,954 0 16 Aug 2021
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 360 631 0 14 Jul 2021
RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su Yu Lu Shengfeng Pan Ahmed Murtadha Bo Wen Yunfeng Liu 278 2,453 0 20 Apr 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 450 2,096 0 31 Dec 2020
On Faithfulness and Factuality in Abstractive Summarization Joshua Maynez Shashi Narayan Bernd Bohnet Ryan T. McDonald HILM 79 1,035 0 02 May 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 439 20,181 0 23 Oct 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Mohammad Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 329 1,904 0 17 Sep 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 659 24,464 0 26 Jul 2019
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 224 1,527 0 24 May 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems Alex Jinpeng Wang Yada Pruksachatkun Nikita Nangia Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 265 2,315 0 02 May 2019
Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization Shashi Narayan Shay B. Cohen Mirella Lapata AILaw 126 1,676 0 27 Aug 2018
QuAC : Question Answering in Context Eunsol Choi He He Mohit Iyyer Mark Yatskar Wen-tau Yih Yejin Choi Percy Liang Luke Zettlemoyer 122 826 0 21 Aug 2018
Know What You Don't Know: Unanswerable Questions for SQuAD Pranav Rajpurkar Robin Jia Percy Liang RALM ELM 279 2,845 0 11 Jun 2018
The NarrativeQA Reading Comprehension Challenge Tomás Kociský Jonathan Richard Schwarz Phil Blunsom Chris Dyer Karl Moritz Hermann Gábor Melis Edward Grefenstette 134 774 0 19 Dec 2017
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 207 2,654 0 09 May 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 524 4,479 0 18 Apr 2017
RACE: Large-scale ReAding Comprehension Dataset From Examinations Guokun Lai Qizhe Xie Hanxiao Liu Yiming Yang Eduard H. Hovy ELM 183 1,348 0 15 Apr 2017
Get To The Point: Summarization with Pointer-Generator Networks A. See Peter J. Liu Christopher D. Manning 3DPC 298 4,019 0 14 Apr 2017
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 347 3,547 0 10 Jun 2015