What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?

12 April 2022

Papers citing "What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?"

50 / 125 papers shown

Title
On the importance of Data Scale in Pretraining Arabic Language Models Abbas Ghaddar Philippe Langlais Mehdi Rezagholizadeh Boxing Chen 27 0 0 15 Jan 2024
Code Security Vulnerability Repair Using Reinforcement Learning with Large Language Models Nafis Tanveer Islam Mohammad Bahrami Karkevandi Peyman Najafirad 32 6 0 13 Jan 2024
xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein Bo Chen Xingyi Cheng Pan Li Yangli-ao Geng Jing Gong ... Chiming Liu Aohan Zeng Yuxiao Dong Jie Tang Leo T. Song 42 101 0 11 Jan 2024
Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation Zhaokun Jiang Ziyin Zhang EGVM 24 3 0 10 Jan 2024
One-Step Diffusion Distillation via Deep Equilibrium Models Zhengyang Geng Ashwini Pokle Trevor Killeen 34 28 0 12 Dec 2023
User Modeling in the Era of Large Language Models: Current Research and Future Directions Zhaoxuan Tan Meng Jiang 30 8 0 11 Dec 2023
Object Recognition as Next Token Prediction Kaiyu Yue Borchun Chen Jonas Geiping Hengduo Li Tom Goldstein Ser-Nam Lim 40 9 0 04 Dec 2023
A Computationally Efficient Sparsified Online Newton Method Fnu Devvrit Sai Surya Duvvuri Rohan Anil Vineet Gupta Cho-Jui Hsieh Inderjit Dhillon 26 0 0 16 Nov 2023
To Transformers and Beyond: Large Language Models for the Genome Micaela Elisa Consens Cameron Dufault Michael Wainberg Duncan Forster Mehran Karimzadeh Hani Goodarzi Fabian J. Theis Alan Moses Bo Wang LM&MA MedIm 26 26 0 13 Nov 2023
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization Tianshi Che Ji Liu Yang Zhou Jiaxiang Ren Jiwen Zhou Victor S. Sheng H. Dai Dejing Dou 30 51 0 23 Oct 2023
InstructDET: Diversifying Referring Object Detection with Generalized Instructions Ronghao Dang Jiangyan Feng Haodong Zhang Chongjian Ge Lin Song ... Chengju Liu Qi Chen Feng Zhu Rui Zhao Yibing Song ObjD 32 11 0 08 Oct 2023
Multiple Physics Pretraining for Physical Surrogate Models Michael McCabe Bruno Régaldo-Saint Blancard Liam Parker Ruben Ohana M. Cranmer ... Francois Lanusse Mariel Pettee Tiberiu Teşileanu Kyunghyun Cho Shirley Ho PINN AI4CE 40 53 0 04 Oct 2023
MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models Kailai Yang Tianlin Zhang Zi-Zhou Kuang Qianqian Xie Jimin Huang Sophia Ananiadou AI4MH 38 47 0 24 Sep 2023
A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models Haoran Xu Young Jin Kim Amr Sharaf Hany Awadalla 41 57 0 20 Sep 2023
Data-Juicer: A One-Stop Data Processing System for Large Language Models Daoyuan Chen Yilun Huang Zhijian Ma Hesen Chen Xuchen Pan ... Zhaoyang Liu Jinyang Gao Yaliang Li Bolin Ding Jingren Zhou SyDa VLM 31 30 0 05 Sep 2023
UnIVAL: Unified Model for Image, Video, Audio and Language Tasks Mustafa Shukor Corentin Dancette Alexandre Ramé Matthieu Cord MoMe MLLM 61 42 0 30 Jul 2023
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets Seonghyeon Ye Doyoung Kim Sungdong Kim Hyeonbin Hwang Seungone Kim Yongrae Jo James Thorne Juho Kim Minjoon Seo ALM 46 99 0 20 Jul 2023
A Comprehensive Overview of Large Language Models Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Naveed Akhtar Nick Barnes Ajmal Mian OffRL 70 529 0 12 Jul 2023
Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models Zaid Alyafeai Maged S. Alshaibani Badr AlKhamissi H. Luqman Ebrahim Alareqi A. Fadel ELM LM&MA AI4MH 22 17 0 28 Jun 2023
Generate to Understand for Representation Changshan Xue Xiande Zhong Xiaoqing Liu VLM 42 0 0 14 Jun 2023
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only Guilherme Penedo Quentin Malartic Daniel Hesslow Ruxandra-Aimée Cojocaru Alessandro Cappelli Hamza Alobeidli B. Pannier Ebtesam Almazrouei Julien Launay 56 751 0 01 Jun 2023
Measuring the Robustness of NLP Models to Domain Shifts Nitay Calderon Naveh Porat Eyal Ben-David Alexander Chapanin Zorik Gekhman Nadav Oved Vitaly Shalumov Roi Reichart 21 7 0 31 May 2023
Large Language Models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery Magdalena Wysocka Oskar Wysocki Maxime Delmas V. Mutel André Freitas LM&MA 35 6 0 28 May 2023
TADA: Task-Agnostic Dialect Adapters for English William B. Held Caleb Ziems Diyi Yang 12 12 0 26 May 2023
Training Transitive and Commutative Multimodal Transformers with LoReTTa Manuel Tran Yashin Dicente Cid Amal Lahiani Fabian J. Theis Tingying Peng Eldad Klaiman 26 2 0 23 May 2023
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity Shayne Longpre Gregory Yauney Emily Reif Katherine Lee Adam Roberts ... Denny Zhou Jason W. Wei Kevin Robinson David M. Mimno Daphne Ippolito 31 149 0 22 May 2023
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages Erik Nijkamp A. Ghobadzadeh Caiming Xiong Silvio Savarese Yingbo Zhou 152 164 0 03 May 2023
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training Nitay Calderon Subhabrata Mukherjee Roi Reichart Amir Kantor 41 17 0 03 May 2023
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond Jingfeng Yang Hongye Jin Ruixiang Tang Xiaotian Han Qizhang Feng Haoming Jiang Bing Yin Xia Hu LM&MA 139 626 0 26 Apr 2023
A Cookbook of Self-Supervised Learning Randall Balestriero Mark Ibrahim Vlad Sobal Ari S. Morcos Shashank Shekhar ... Pierre Fernandez Amir Bar Hamed Pirsiavash Yann LeCun Micah Goldblum SyDa FedML SSL 50 274 0 24 Apr 2023
nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales Yiqun Yao Siqi Fan Xiusheng Huang Xuezhi Fang Xiang Li ... Peng Han Shuo Shang Kang Liu Aixin Sun Yequan Wang 33 6 0 14 Apr 2023
A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques Wenbin Li Hakim Hacid Ebtesam Almazrouei Merouane Debbah 34 13 0 16 Feb 2023
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning Shayne Longpre Le Hou Tu Vu Albert Webson Hyung Won Chung ... Denny Zhou Quoc V. Le Barret Zoph Jason W. Wei Adam Roberts ALM 41 633 0 31 Jan 2023
Cramming: Training a Language Model on a Single GPU in One Day Jonas Geiping Tom Goldstein MoE 30 85 0 28 Dec 2022
SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning M Saiful Bari Aston Zhang Shuai Zheng Xingjian Shi Yi Zhu Chenyu You Mu Li RALM VLM VPVLM LRM 48 5 0 21 Dec 2022
HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation Hamish Ivison Akshita Bhagia Yizhong Wang Hannaneh Hajishirzi Matthew E. Peters 48 16 0 20 Dec 2022
BARTSmiles: Generative Masked Language Models for Molecular Representations Gayane Chilingaryan Hovhannes Tamoyan Ani Tevosyan N. Babayan L. Khondkaryan Karen Hambardzumyan Zaven Navoyan Hrant Khachatrian Armen Aghajanyan SSL 35 25 0 29 Nov 2022
On the Compositional Generalization Gap of In-Context Learning Arian Hosseini Ankit Vani Dzmitry Bahdanau Alessandro Sordoni Rameswar Panda 27 24 0 15 Nov 2022
Large Language Models Struggle to Learn Long-Tail Knowledge Nikhil Kandpal H. Deng Adam Roberts Eric Wallace Colin Raffel RALM KELM 56 388 0 15 Nov 2022
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 118 2,315 0 09 Nov 2022
GPS: Genetic Prompt Search for Efficient Few-shot Learning Hanwei Xu Yujun Chen Yulun Du Nan Shao Yanggang Wang Haiyu Li Zhilin Yang VLM 14 28 0 31 Oct 2022
What Language Model to Train if You Have One Million GPU Hours? Teven Le Scao Thomas Wang Daniel Hesslow Lucile Saulnier Stas Bekman ... Lintang Sutawika Jaesung Tae Zheng-Xin Yong Julien Launay Iz Beltagy MoE AI4CE 230 103 0 27 Oct 2022
Can language models handle recursively nested grammatical structures? A case study on comparing models and humans Andrew Kyle Lampinen ReLM ELM 27 35 0 27 Oct 2022
Scaling Laws Beyond Backpropagation Matthew J. Filipovich Alessandro Cappelli Daniel Hesslow Julien Launay 19 3 0 26 Oct 2022
Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models Hao Liu Xinyang Geng Lisa Lee Igor Mordatch Sergey Levine Sharan Narang Pieter Abbeel KELM CLL 35 2 0 24 Oct 2022
WikiWhy: Answering and Explaining Cause-and-Effect Questions Matthew Ho Aditya Sharma Justin Chang Michael Stephen Saxon Sharon Levy Yujie Lu William Yang Wang ReLM KELM LRM 64 16 0 21 Oct 2022
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 100 2,999 0 20 Oct 2022
Transcending Scaling Laws with 0.1% Extra Compute Yi Tay Jason W. Wei Hyung Won Chung Vinh Q. Tran David R. So ... Donald Metzler Slav Petrov N. Houlsby Quoc V. Le Mostafa Dehghani LRM 44 68 0 20 Oct 2022
Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization Yuxian Gu Pei Ke Xiaoyan Zhu Minlie Huang ALM 39 17 0 17 Oct 2022
VIMA: General Robot Manipulation with Multimodal Prompts Yunfan Jiang Agrim Gupta Zichen Zhang Guanzhi Wang Yongqiang Dou Yanjun Chen Li Fei-Fei Anima Anandkumar Yuke Zhu Linxi Fan LM&Ro 33 336 0 06 Oct 2022