Large Language Model Confidence Estimation via Black-Box Access

21 February 2025

Papers citing "Large Language Model Confidence Estimation via Black-Box Access"

40 / 40 papers shown

Title
Towards Harmonized Uncertainty Estimation for Large Language Models Rui Li Jing Long Muge Qi Heming Xia Lei Sha Peiyi Wang Zhifang Sui UQCV 54 0 0 25 May 2025
A Survey of Calibration Process for Black-Box LLMs Liangru Xie Hui Liu Jingying Zeng Xianfeng Tang Yan Han Chen Luo Jing Huang Zhen Li Suhang Wang Qi He 125 4 0 17 Dec 2024
Black-box Uncertainty Quantification Method for LLM-as-a-Judge Nico Wagner Michael Desmond Rahul Nair Zahra Ashktorab Elizabeth M. Daly Qian Pan Martin Santillan Cooper James M. Johnson Werner Geyer ELM UQCV 68 5 0 15 Oct 2024
Thermometer: Towards Universal Calibration for Large Language Models Maohao Shen Subhro Das Kristjan Greenewald P. Sattigeri Greg Wornell Soumya Ghosh 96 11 0 20 Feb 2024
Benchmarking LLMs via Uncertainty Quantification Fanghua Ye Mingming Yang Jianhui Pang Longyue Wang Derek F. Wong Emine Yilmaz Shuming Shi Zhaopeng Tu ELM 213 59 0 23 Jan 2024
A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models S.M. Towhidul Islam Tonmoy S. M. M. Zaman Vinija Jain Anku Rani Vipula Rawte Aman Chadha Amitava Das HILM 105 206 0 02 Jan 2024
Retrieval-Augmented Generation for Large Language Models: A Survey Yunfan Gao Yun Xiong Xinyu Gao Kangxiang Jia Jinliu Pan Yuxi Bi Yi Dai Jiawei Sun Meng Wang Haofen Wang 3DV RALM 219 1,814 1 18 Dec 2023
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs Miao Xiong Zhiyuan Hu Xinyang Lu Yifei Li Jie Fu Junxian He Bryan Hooi 208 447 0 22 Jun 2023
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models Zhen Lin Shubhendu Trivedi Jimeng Sun HILM 187 153 0 30 May 2023
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback Katherine Tian E. Mitchell Allan Zhou Archit Sharma Rafael Rafailov Huaxiu Yao Chelsea Finn Christopher D. Manning 126 354 0 24 May 2023
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation Lorenz Kuhn Y. Gal Sebastian Farquhar UQLM 204 308 0 19 Feb 2023
A Close Look into the Calibration of Pre-trained Language Models Yangyi Chen Lifan Yuan Ganqu Cui Zhiyuan Liu Heng Ji 131 51 0 31 Oct 2022
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis Yuxin Xiao Paul Pu Liang Umang Bhatt Willie Neiswanger Ruslan Salakhutdinov Louis-Philippe Morency 222 96 0 10 Oct 2022
Language Models (Mostly) Know What They Know Saurav Kadavath Tom Conerly Amanda Askell T. Henighan Dawn Drain ... Nicholas Joseph Benjamin Mann Sam McCandlish C. Olah Jared Kaplan ELM 122 830 0 11 Jul 2022
Teaching Models to Express Their Uncertainty in Words Stephanie C. Lin Jacob Hilton Owain Evans OOD 94 423 0 28 May 2022
UL2: Unifying Language Learning Paradigms Yi Tay Mostafa Dehghani Vinh Q. Tran Xavier Garcia Jason W. Wei ... Tal Schuster H. Zheng Denny Zhou N. Houlsby Donald Metzler AI4CE 121 313 0 10 May 2022
On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency Seohong Park Cornelia Caragea UQCV 49 36 0 14 Mar 2022
A Survey of Uncertainty in Deep Neural Networks J. Gawlikowski Cedrique Rovile Njieutcheu Tassi Mohsin Ali Jongseo Lee Matthias Humt ... R. Roscher Muhammad Shahzad Wen Yang R. Bamler Xiaoxiang Zhu BDL UQCV OOD 235 1,155 0 07 Jul 2021
Knowing More About Questions Can Help: Improving Calibration in Question Answering Shujian Zhang Chengyue Gong Eunsol Choi UQLM 87 58 0 02 Jun 2021
What Are Bayesian Neural Network Posteriors Really Like? Pavel Izmailov Sharad Vikram Matthew D. Hoffman A. Wilson UQCV BDL 74 387 0 29 Apr 2021
Reducing conversational agents' overconfidence through linguistic calibration Sabrina J. Mielke Arthur Szlam Emily Dinan Y-Lan Boureau 280 169 0 30 Dec 2020
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering Zhengbao Jiang Jun Araki Haibo Ding Graham Neubig UQCV 60 436 0 02 Dec 2020
DeBERTa: Decoding-enhanced BERT with Disentangled Attention Pengcheng He Xiaodong Liu Jianfeng Gao Weizhu Chen AAML 165 2,750 0 05 Jun 2020
Calibration of Pre-trained Transformers Shrey Desai Greg Durrett UQLM 296 301 0 17 Mar 2020
Calibrating Deep Neural Networks using Focal Loss Jishnu Mukhoti Viveka Kulharia Amartya Sanyal Stuart Golodetz Philip Torr P. Dokania UQCV 85 465 0 21 Feb 2020
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization Jingqing Zhang Yao-Min Zhao Mohammad Saleh Peter J. Liu RALM 3DGS 297 2,051 0 18 Dec 2019
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension M. Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdel-rahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer AIMat VLM 264 10,851 0 29 Oct 2019
Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration Meelis Kull Miquel Perelló Nieto Markus Kängsepp Telmo de Menezes e Silva Filho Hao Song Peter A. Flach UQCV 81 382 0 28 Oct 2019
The Curious Case of Neural Text Degeneration Ari Holtzman Jan Buys Li Du Maxwell Forbes Yejin Choi 199 3,210 0 22 Apr 2019
Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization Shashi Narayan Shay B. Cohen Mirella Lapata AILaw 146 1,683 0 27 Aug 2018
CoQA: A Conversational Question Answering Challenge Siva Reddy Danqi Chen Christopher D. Manning RALM HAI 114 1,210 0 21 Aug 2018
mixup: Beyond Empirical Risk Minimization Hongyi Zhang Moustapha Cissé Yann N. Dauphin David Lopez-Paz NoLa 289 9,803 0 25 Oct 2017
On Calibration of Modern Neural Networks Chuan Guo Geoff Pleiss Yu Sun Kilian Q. Weinberger UQCV 299 5,862 0 14 Jun 2017
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 231 2,686 0 09 May 2017
Get To The Point: Summarization with Pointer-Generator Networks A. See Peter J. Liu Christopher D. Manning 3DPC 311 4,026 0 14 Apr 2017
Regularizing Neural Networks by Penalizing Confident Output Distributions Gabriel Pereyra George Tucker J. Chorowski Lukasz Kaiser Geoffrey E. Hinton NoLa 165 1,141 0 23 Jan 2017
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles Balaji Lakshminarayanan Alexander Pritzel Charles Blundell UQCV BDL 842 5,841 0 05 Dec 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 316 8,169 0 16 Jun 2016
Rethinking the Inception Architecture for Computer Vision Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Z. Wojna 3DV BDL 886 27,416 0 02 Dec 2015
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 351 3,552 0 10 Jun 2015