On the Role of Bidirectionality in Language Model Pre-Training

On the Role of Bidirectionality in Language Model Pre-Training

24 May 2022

Luke Zettlemoyer

Papers citing "On the Role of Bidirectionality in Language Model Pre-Training"

14 / 14 papers shown

Title
The pitfalls of next-token prediction Gregor Bachmann Vaishnavh Nagarajan 37 62 0 11 Mar 2024
When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality Brielen Madureira Patrick Kahardipraja David Schlangen 39 2 0 20 Feb 2024
On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Research Made Nindyatama Nityasya Haryo Akbarianto Wibowo Alham Fikri Aji Genta Indra Winata Radityo Eko Prasojo Phil Blunsom A. Kuncoro 19 8 0 05 Jun 2023
Training Transitive and Commutative Multimodal Transformers with LoReTTa Manuel Tran Yashin Dicente Cid Amal Lahiani Fabian J. Theis Tingying Peng Eldad Klaiman 26 2 0 23 May 2023
A Cookbook of Self-Supervised Learning Randall Balestriero Mark Ibrahim Vlad Sobal Ari S. Morcos Shashank Shekhar ... Pierre Fernandez Amir Bar Hamed Pirsiavash Yann LeCun Micah Goldblum SyDa FedML SSL 44 273 0 24 Apr 2023
Language Model Behavior: A Comprehensive Survey Tyler A. Chang Benjamin Bergen VLM LRM LM&MA 27 103 0 20 Mar 2023
Cramming: Training a Language Model on a Single GPU in One Day Jonas Geiping Tom Goldstein MoE 30 84 0 28 Dec 2022
Training Trajectories of Language Models Across Scales Mengzhou Xia Mikel Artetxe Chunting Zhou Xi Lin Ramakanth Pasunuru Danqi Chen Luke Zettlemoyer Ves Stoyanov AIFin LRM 31 53 0 19 Dec 2022
Nonparametric Masked Language Modeling Sewon Min Weijia Shi M. Lewis Xilun Chen Wen-tau Yih Hannaneh Hajishirzi Luke Zettlemoyer RALM 48 48 0 02 Dec 2022
Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models Hao Liu Xinyang Geng Lisa Lee Igor Mordatch Sergey Levine Sharan Narang Pieter Abbeel KELM CLL 33 2 0 24 Oct 2022
Efficient Training of Language Models to Fill in the Middle Mohammad Bavarian Heewoo Jun Nikolas Tezak John Schulman C. McLeavey Jerry Tworek Mark Chen 6 179 0 28 Jul 2022
Language Models are General-Purpose Interfaces Y. Hao Haoyu Song Li Dong Shaohan Huang Zewen Chi Wenhui Wang Shuming Ma Furu Wei MLLM 30 95 0 13 Jun 2022
Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey Bonan Min Hayley L Ross Elior Sulem Amir Pouran Ben Veyseh Thien Huu Nguyen Oscar Sainz Eneko Agirre Ilana Heinz Dan Roth LM&MA VLM AI4CE 83 1,030 0 01 Nov 2021
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018