How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances

11 October 2023

Zihan Zhang

Meng Fang

Lingxi Chen

Mohammad-Reza Namazi-Rad

Jun Wang

KELM

ArXiv (abs)PDF HTML Github (134★)

Papers citing "How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances"

50 / 151 papers shown

Title
Quantifying Memorization Across Neural Language Models Nicholas Carlini Daphne Ippolito Matthew Jagielski Katherine Lee Florian Tramèr Chiyuan Zhang PILM 124 630 0 15 Feb 2022
Locating and Editing Factual Associations in GPT Kevin Meng David Bau A. Andonian Yonatan Belinkov KELM 251 1,389 0 10 Feb 2022
Survey of Hallucination in Natural Language Generation Ziwei Ji Nayeon Lee Rita Frieske Tiezheng Yu D. Su ... Delong Chen Wenliang Dai Ho Shu Chan Andrea Madotto Pascale Fung HILM LRM 236 2,435 0 08 Feb 2022
Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval Uri Alon Frank F. Xu Junxian He Sudipta Sengupta Dan Roth Graham Neubig RALM 127 63 0 28 Jan 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 853 9,714 0 28 Jan 2022
WebGPT: Browser-assisted question-answering with human feedback Reiichiro Nakano Jacob Hilton S. Balaji Jeff Wu Ouyang Long ... Gretchen Krueger Kevin Button Matthew Knight B. Chess John Schulman ALM RALM 196 1,294 0 17 Dec 2021
Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedback Niket Tandon Aman Madaan Peter Clark Yiming Yang KELM 67 40 0 16 Dec 2021
Improving language models by retrieving from trillions of tokens Sebastian Borgeaud A. Mensch Jordan Hoffmann Trevor Cai Eliza Rutherford ... Simon Osindero Karen Simonyan Jack W. Rae Erich Elsen Laurent Sifre KELM RALM 254 1,100 0 08 Dec 2021
Time Waits for No One! Analysis and Challenges of Temporal Misalignment Kelvin Luu Daniel Khashabi Suchin Gururangan Karishma Mandyam Noah A. Smith 99 91 0 14 Nov 2021
Fast Model Editing at Scale E. Mitchell Charles Lin Antoine Bosselut Chelsea Finn Christopher D. Manning KELM 344 378 0 21 Oct 2021
GNN-LM: Language Modeling based on Global Contexts via GNN Yuxian Meng Shi Zong Xiaoya Li Xiaofei Sun Tianwei Zhang Leilei Gan Jiwei Li LRM 90 39 0 17 Oct 2021
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora Xisen Jin Dejiao Zhang Henghui Zhu Wei Xiao Shang-Wen Li Xiaokai Wei Andrew O. Arnold Xiang Ren KELM CLL 106 117 0 16 Oct 2021
Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey Xiaokai Wei Shen Wang Dejiao Zhang Parminder Bhatia Andrew O. Arnold KELM 93 46 0 16 Oct 2021
Towards Continual Knowledge Learning of Language Models Joel Jang Seonghyeon Ye Sohee Yang Joongbo Shin Janghoon Han Gyeonghun Kim Stanley Jungkyu Choi Minjoon Seo CLL KELM 301 161 0 07 Oct 2021
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief Nora Kassner Oyvind Tafjord Hinrich Schütze Peter Clark KELM LRM 295 65 0 29 Sep 2021
Efficient Nearest Neighbor Language Models Junxian He Graham Neubig Taylor Berg-Kirkpatrick RALM 233 104 0 09 Sep 2021
DEMix Layers: Disentangling Domains for Modular Language Modeling Suchin Gururangan Michael Lewis Ari Holtzman Noah A. Smith Luke Zettlemoyer KELM MoE 101 136 0 11 Aug 2021
Internet-Augmented Dialogue Generation M. Komeili Kurt Shuster Jason Weston RALM 300 289 0 15 Jul 2021
Time-Aware Language Models as Temporal Knowledge Bases Bhuwan Dhingra Jeremy R. Cole Julian Martin Eisenschlos D. Gillick Jacob Eisenstein William W. Cohen KELM 101 281 0 29 Jun 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 502 10,526 0 17 Jun 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 339 682 0 21 Apr 2021
Knowledge Neurons in Pretrained Transformers Damai Dai Li Dong Y. Hao Zhifang Sui Baobao Chang Furu Wei KELM MU 113 465 0 18 Apr 2021
Editing Factual Knowledge in Language Models Nicola De Cao Wilker Aziz Ivan Titov KELM 127 513 0 16 Apr 2021
Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media Paul Röttger J. Pierrehumbert 98 68 0 16 Apr 2021
Mind the Gap: Assessing Temporal Generalization in Neural Language Models Angeliki Lazaridou A. Kuncoro E. Gribovskaya Devang Agrawal Adam Liska ... Sebastian Ruder Dani Yogatama Kris Cao Susannah Young Phil Blunsom VLM 134 219 0 03 Feb 2021
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering Fengbin Zhu Wenqiang Lei Chao Wang Jianming Zheng Soujanya Poria Tat-Seng Chua RALM 252 257 0 04 Jan 2021
Transformer Feed-Forward Layers Are Key-Value Memories Mor Geva R. Schuster Jonathan Berant Omer Levy KELM 182 848 0 29 Dec 2020
Continual Lifelong Learning in Natural Language Processing: A Survey Magdalena Biesialska Katarzyna Biesialska Marta R. Costa-jussá KELM CLL 90 222 0 17 Dec 2020
Modifying Memories in Transformer Models Chen Zhu A. S. Rawat Manzil Zaheer Srinadh Bhojanapalli Daliang Li Felix X. Yu Sanjiv Kumar KELM 118 203 0 01 Dec 2020
A Survey of Knowledge-Enhanced Text Generation Wenhao Yu Chenguang Zhu Zaitang Li Zhiting Hu Qingyun Wang Heng Ji Meng Jiang 126 288 0 09 Oct 2020
Rewriting a Deep Generative Model David Bau Steven Liu Tongzhou Wang Jun-Yan Zhu Antonio Torralba GAN DRL 103 140 0 30 Jul 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 904 42,463 0 28 May 2020
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting Sanyuan Chen Yutai Hou Yiming Cui Wanxiang Che Ting Liu Xiangzhan Yu KELM CLL 120 225 0 27 Apr 2020
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks Suchin Gururangan Ana Marasović Swabha Swayamdipta Kyle Lo Iz Beltagy Doug Downey Noah A. Smith VLM AI4CE CLL 167 2,440 0 23 Apr 2020
Dense Passage Retrieval for Open-Domain Question Answering Vladimir Karpukhin Barlas Oğuz Sewon Min Patrick Lewis Ledell Yu Wu Sergey Edunov Danqi Chen Wen-tau Yih RALM 207 3,799 0 10 Apr 2020
Editable Neural Networks A. Sinitsin Vsevolod Plokhotnyuk Dmitriy V. Pyrkin Sergei Popov Artem Babenko KELM 113 182 0 01 Apr 2020
How Much Knowledge Can You Pack Into the Parameters of a Language Model? Adam Roberts Colin Raffel Noam M. Shazeer KELM 136 897 0 10 Feb 2020
REALM: Retrieval-Augmented Language Model Pre-Training Kelvin Guu Kenton Lee Zora Tung Panupong Pasupat Ming-Wei Chang RALM 147 2,121 0 10 Feb 2020
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters Ruize Wang Duyu Tang Nan Duan Zhongyu Wei Xuanjing Huang Jianshu Ji Guihong Cao Daxin Jiang Ming Zhou KELM 129 556 0 05 Feb 2020
A Survey on Knowledge Graphs: Representation, Acquisition and Applications Shaoxiong Ji Shirui Pan Min Zhang Pekka Marttinen Philip S. Yu 300 2,004 0 02 Feb 2020
How Can We Know What Language Models Know? Zhengbao Jiang Frank F. Xu Jun Araki Graham Neubig KELM 146 1,412 0 28 Nov 2019
Generalization through Memorization: Nearest Neighbor Language Models Urvashi Khandelwal Omer Levy Dan Jurafsky Luke Zettlemoyer M. Lewis RALM 177 842 0 01 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 503 20,342 0 23 Oct 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 587 2,677 0 03 Sep 2019
Parameter-Efficient Transfer Learning for NLP N. Houlsby A. Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin de Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly 223 4,529 0 02 Feb 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,324 0 11 Oct 2018
Adaptive Input Representations for Neural Language Modeling Alexei Baevski Michael Auli 111 390 0 28 Sep 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 805 132,725 0 12 Jun 2017
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks Chelsea Finn Pieter Abbeel Sergey Levine OOD 833 11,961 0 09 Mar 2017
Overcoming catastrophic forgetting in neural networks J. Kirkpatrick Razvan Pascanu Neil C. Rabinowitz J. Veness Guillaume Desjardins ... A. Grabska-Barwinska Demis Hassabis Claudia Clopath D. Kumaran R. Hadsell CLL 374 7,587 0 02 Dec 2016