How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering

2 December 2020

Graham Neubig

Papers citing "How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering"

30 / 80 papers shown

Title
Calibration Error Estimation Using Fuzzy Binning Geetanjali Bihani Julia Taylor Rayz 97 2 0 30 Apr 2023
Context-Aware Differential Privacy for Language Modeling M. H. Dinh Ferdinando Fioretto 25 2 0 28 Jan 2023
Automated Identification of Eviction Status from Electronic Health Record Notes Zonghai Yao J. Tsai Weisong Liu David Levy Emily Druhl J. Reisman Hongfeng Yu 22 10 0 06 Dec 2022
SPE: Symmetrical Prompt Enhancement for Fact Probing Yiyuan Li Tong Che Yezhen Wang Zhengbao Jiang Caiming Xiong Snigdha Chaturvedi 31 6 0 14 Nov 2022
Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates Dongfang Li Baotian Hu Qingcai Chen 16 8 0 06 Nov 2022
The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation Barbara Plank 30 97 0 04 Nov 2022
Stop Measuring Calibration When Humans Disagree Joris Baan Wilker Aziz Barbara Plank Raquel Fernández 24 53 0 28 Oct 2022
Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity Dennis Ulmer J. Frellsen Christian Hardmeier 189 22 0 20 Oct 2022
NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly Yi Ren Fung Tuhin Chakraborty Hao Guo Owen Rambow Smaranda Muresan Heng Ji 21 39 0 16 Oct 2022
CHARD: Clinical Health-Aware Reasoning Across Dimensions for Text Generation Models Steven Y. Feng Vivek Khetan Bogdan Sacaleanu A. Gershman Eduard H. Hovy LRM 35 10 0 09 Oct 2022
Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation Xingdi Yuan Tong Wang Yen-Hsiang Wang Emery Fine Rania Abdelghani Pauline Lucas Hélene Sauzéon Pierre-Yves Oudeyer 30 29 0 22 Sep 2022
On the Relation between Sensitivity and Accuracy in In-context Learning Yanda Chen Chen Zhao Zhou Yu Kathleen McKeown He He 182 77 0 16 Sep 2022
Meaning without reference in large language models S. Piantadosi Felix Hill 22 72 0 05 Aug 2022
Language models show human-like content effects on reasoning tasks Ishita Dasgupta Andrew Kyle Lampinen Stephanie C. Y. Chan Hannah R. Sheahan Antonia Creswell D. Kumaran James L. McClelland Felix Hill ReLM LRM 30 181 0 14 Jul 2022
Language Models (Mostly) Know What They Know Saurav Kadavath Tom Conerly Amanda Askell T. Henighan Dawn Drain ... Nicholas Joseph Benjamin Mann Sam McCandlish C. Olah Jared Kaplan ELM 47 712 0 11 Jul 2022
Re-Examining Calibration: The Case of Question Answering Chenglei Si Chen Zhao Sewon Min Jordan L. Boyd-Graber 61 30 0 25 May 2022
Learning a Better Initialization for Soft Prompts via Meta-Learning Yukun Huang Kun Qian Zhou Yu VLM 47 9 0 25 May 2022
Life after BERT: What do Other Muppets Understand about Language? Vladislav Lialin Kevin Zhao Namrata Shivagunde Anna Rumshisky 44 6 0 21 May 2022
Calibration of Natural Language Understanding Models with Venn--ABERS Predictors Patrizio Giovannotti 38 6 0 21 May 2022
Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction Hui-Syuan Yeh Thomas Lavergne Pierre Zweigenbaum 21 10 0 21 Apr 2022
Uncertainty Estimation for Language Reward Models Adam Gleave G. Irving UQLM 36 31 0 14 Mar 2022
Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings Neeraj Varshney Swaroop Mishra Chitta Baral 13 55 0 01 Mar 2022
Semantic-Oriented Unlabeled Priming for Large-Scale Language Models Yanchen Liu Timo Schick Hinrich Schütze VLM 28 15 0 12 Feb 2022
What Has Been Enhanced in my Knowledge-Enhanced Language Model? Yifan Hou Guoji Fu Mrinmaya Sachan KELM 33 1 0 02 Feb 2022
Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey Bonan Min Hayley L Ross Elior Sulem Amir Pouran Ben Veyseh Thien Huu Nguyen Oscar Sainz Eneko Agirre Ilana Heinz Dan Roth LM&MA VLM AI4CE 83 1,030 0 01 Nov 2021
Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units Anurag Katakkar A. Black AuLLM 27 1 0 31 Oct 2021
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing Pengfei Liu Weizhe Yuan Jinlan Fu Zhengbao Jiang Hiroaki Hayashi Graham Neubig VLM SyDa 37 3,831 0 28 Jul 2021
Calibration of Pre-trained Transformers Shrey Desai Greg Durrett UQLM 243 289 0 17 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 261 4,489 0 23 Jan 2020
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 417 2,588 0 03 Sep 2019