Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models

24 May 2023

Amirhossein Kazemnejad

Mehdi Rezagholizadeh

Prasanna Parthasarathi

Sarath Chandar

ELM

ArXiv (abs)PDF HTML

Papers citing "Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models"

33 / 33 papers shown

Title
StarCoder: may the source be with you! Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov ... Sean M. Hughes Thomas Wolf Arjun Guha Leandro von Werra H. D. Vries 129 789 0 09 May 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.5K 13,490 0 27 Feb 2023
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? Chengwei Qin Aston Zhang Zhuosheng Zhang Jiaao Chen Michihiro Yasunaga Diyi Yang LM&MA AI4MH LRM ELM 166 702 0 08 Feb 2023
Prompting Language Models for Linguistic Structure Terra Blevins Hila Gonen Luke Zettlemoyer LRM 120 44 0 15 Nov 2022
Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversation Yanyang Li Jianqiao Zhao Michael R. Lyu Liwei Wang 60 16 0 03 Nov 2022
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 234 3,165 0 20 Oct 2022
Generate rather than Retrieve: Large Language Models are Strong Context Generators Wenhao Yu Dan Iter Shuohang Wang Yichong Xu Mingxuan Ju Soumya Sanyal Chenguang Zhu Michael Zeng Meng Jiang RALM AIMat 347 339 0 21 Sep 2022
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 292 2,521 0 15 Jun 2022
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 371 3,700 0 02 May 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 537 6,301 0 05 Apr 2022
Multi-Stage Prompting for Knowledgeable Dialogue Generation Zihan Liu M. Patwary R. Prenger Shrimai Prabhumoye Ming-Yu Liu Mohammad Shoeybi Bryan Catanzaro 58 50 0 16 Mar 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 894 13,228 0 04 Mar 2022
Schrödinger's Tree -- On Syntax and Neural Language Models Artur Kulmizev Joakim Nivre 70 6 0 17 Oct 2021
Generated Knowledge Prompting for Commonsense Reasoning Jiacheng Liu Alisa Liu Ximing Lu Sean Welleck Peter West Ronan Le Bras Yejin Choi Hannaneh Hajishirzi KELM RALM ReLM LLMAG LRM 139 326 0 15 Oct 2021
Finetuned Language Models Are Zero-Shot Learners Jason W. Wei Maarten Bosma Vincent Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai Quoc V. Le ALM UQCV 254 3,789 0 03 Sep 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 238 5,675 0 07 Jul 2021
A Closer Look at How Fine-tuning Changes BERT Yichu Zhou Vivek Srikumar 67 68 0 27 Jun 2021
Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA? Cunxiang Wang Pai Liu Yue Zhang RALM 104 84 0 03 Jun 2021
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts Guanghui Qin J. Eisner 65 549 0 14 Apr 2021
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 904 42,520 0 28 May 2020
What Happens To BERT Embeddings During Fine-tuning? Amil Merchant Elahe Rahimtoroghi Ellie Pavlick Ian Tenney 95 188 0 29 Apr 2020
Dense Passage Retrieval for Open-Domain Question Answering Vladimir Karpukhin Barlas Oğuz Sewon Min Patrick Lewis Ledell Yu Wu Sergey Edunov Danqi Chen Wen-tau Yih RALM 209 3,799 0 10 Apr 2020
How Much Knowledge Can You Pack Into the Parameters of a Language Model? Adam Roberts Colin Raffel Noam M. Shazeer KELM 138 897 0 10 Feb 2020
How Can We Know What Language Models Know? Zhengbao Jiang Frank F. Xu Jun Araki Graham Neubig KELM 149 1,412 0 28 Nov 2019
Evaluating Commonsense in Pre-trained Language Models Xuhui Zhou Yue Zhang Leyang Cui Dandan Huang AI4MH LRM 78 185 0 27 Nov 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 587 2,680 0 03 Sep 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 700 24,572 0 26 Jul 2019
What do you learn from context? Probing for sentence structure in contextualized word representations Ian Tenney Patrick Xia Berlin Chen Alex Jinpeng Wang Adam Poliak ... Najoung Kim Benjamin Van Durme Samuel R. Bowman Dipanjan Das Ellie Pavlick 189 865 0 15 May 2019
BERT Rediscovers the Classical NLP Pipeline Ian Tenney Dipanjan Das Ellie Pavlick MILM SSeg 145 1,482 0 15 May 2019
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference R. Thomas McCoy Ellie Pavlick Tal Linzen 145 1,244 0 04 Feb 2019
Passage Re-ranking with BERT Rodrigo Nogueira Kyunghyun Cho OOD 128 1,098 0 13 Jan 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,324 0 11 Oct 2018
Representation Learning with Contrastive Predictive Coding Aaron van den Oord Yazhe Li Oriol Vinyals DRL SSL 356 10,369 0 10 Jul 2018