Prediction-Powered Ranking of Large Language Models

27 February 2024

Ivi Chatzi

Eleni Straitouri

Suhas Thejaswi

Manuel Gomez Rodriguez

ALM

ArXiv PDF HTML

Papers citing "Prediction-Powered Ranking of Large Language Models"

45 / 45 papers shown

Title
Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators Yinhong Liu Han Zhou Zhijiang Guo Ehsan Shareghi Ivan Vulić Anna Korhonen Nigel Collier ALM 160 77 0 20 Jan 2025
PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations Ruosen Li Teerth Patel Xinya Du LLMAG ALM 109 100 0 03 Jan 2025
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data Florian E. Dorner Vivian Y. Nastl Moritz Hardt ELM ALM 64 8 0 17 Oct 2024
Can Unconfident LLM Annotations Be Used for Confident Conclusions? Kristina Gligorić Tijana Zrnic Cinoo Lee Emmanuel J. Candès Dan Jurafsky 98 8 0 27 Aug 2024
AutoEval Done Right: Using Synthetic Data for Model Evaluation Pierre Boyeau Anastasios Nikolas Angelopoulos N. Yosef Jitendra Malik Michael I. Jordan SyDa 54 20 0 09 Mar 2024
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference Wei-Lin Chiang Lianmin Zheng Ying Sheng Anastasios Nikolas Angelopoulos Tianle Li ... Hao Zhang Banghua Zhu Michael I. Jordan Joseph E. Gonzalez Ion Stoica OSLM 91 536 0 07 Mar 2024
Evaluating Language Model Agency through Negotiations Tim R. Davidson V. Veselovsky Martin Josifoski Maxime Peyrard Antoine Bosselut Michal Kosinski Robert West LLMAG 52 24 0 09 Jan 2024
Elo Uncovered: Robustness and Best Practices in Language Model Evaluation M. Boubdir Edward Kim Beyza Ermis Sara Hooker Marzieh Fadaee ELM 44 36 0 29 Nov 2023
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems Jon Saad-Falcon Omar Khattab Christopher Potts Matei A. Zaharia RALM 61 112 0 16 Nov 2023
PPI++: Efficient Prediction-Powered Inference Anastasios Nikolas Angelopoulos John C. Duchi Tijana Zrnic 104 44 0 02 Nov 2023
Generative Judge for Evaluating Alignment Junlong Li Shichao Sun Weizhe Yuan Run-Ze Fan Hai Zhao Pengfei Liu ELM ALM 44 81 0 09 Oct 2023
Cross-Prediction-Powered Inference Tijana Zrnic Emmanuel J. Candès 112 28 0 28 Sep 2023
Large language models can accurately predict searcher preferences Paul Thomas S. Spielman Nick Craswell Bhaskar Mitra ALM LRM 49 147 0 19 Sep 2023
Confident Feature Ranking Bitya Neuhof Y. Benjamini FAtt 46 3 0 28 Jul 2023
LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models Adian Liusie Potsawee Manakul Mark Gales ELM 44 36 0 15 Jul 2023
A Survey on Evaluation of Large Language Models Yu-Chu Chang Xu Wang Jindong Wang Yuanyi Wu Linyi Yang ... Yue Zhang Yi-Ju Chang Philip S. Yu Qian Yang Xingxu Xie ELM LM&MA ALM 97 1,619 0 06 Jul 2023
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting Zhen Qin R. Jagerman Kai Hui Honglei Zhuang Junru Wu ... Tianqi Liu Jialu Liu Donald Metzler Xuanhui Wang Michael Bendersky ALM RALM 69 235 0 30 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 236 4,186 0 09 Jun 2023
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization Yidong Wang Zhuohao Yu Zhengran Zeng Linyi Yang Cunxiang Wang ... Jindong Wang Xingxu Xie Wei Ye Shi-Bo Zhang Yue Zhang ALM ELM 85 242 0 08 Jun 2023
Benchmarking Foundation Models with Language-Model-as-an-Examiner Yushi Bai Jiahao Ying Yixin Cao Xin Lv Yuze He ... Yijia Xiao Haozhe Lyu Jiayin Zhang Juanzi Li Lei Hou ALM ELM 58 141 0 07 Jun 2023
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion Dongfu Jiang Xiang Ren Bill Yuchen Lin ELM 45 303 0 05 Jun 2023
Large Language Models are not Fair Evaluators Peiyi Wang Lei Li Liang Chen Zefan Cai Dawei Zhu Binghuai Lin Yunbo Cao Qi Liu Tianyu Liu Zhifang Sui ALM 89 542 0 29 May 2023
QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers Artidoro Pagnoni Ari Holtzman Luke Zettlemoyer ALM 110 2,454 0 23 May 2023
LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models Yen-Ting Lin Yun-Nung Chen 38 92 0 23 May 2023
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback Yann Dubois Xuechen Li Rohan Taori Tianyi Zhang Ishaan Gulrajani Jimmy Ba Carlos Guestrin Percy Liang Tatsunori B. Hashimoto ALM 97 581 0 22 May 2023
Large Language Models are Zero-Shot Rankers for Recommender Systems Yupeng Hou Junjie Zhang Zihan Lin Hongyu Lu Ruobing Xie Julian McAuley Wayne Xin Zhao LRM ALM 54 311 0 15 May 2023
Can Large Language Models Be an Alternative to Human Evaluations? Cheng-Han Chiang Hung-yi Lee ALM LM&MA 245 596 0 03 May 2023
Human-like Summarization Evaluation with ChatGPT Mingqi Gao Jie Ruan Renliang Sun Xunjian Yin Shiping Yang Xiaojun Wan ALM AI4MH 35 128 0 05 Apr 2023
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? Chengwei Qin Aston Zhang Zhuosheng Zhang Jiaao Chen Michihiro Yasunaga Diyi Yang LM&MA AI4MH LRM ELM 122 689 0 08 Feb 2023
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning Shayne Longpre Le Hou Tu Vu Albert Webson Hyung Won Chung ... Denny Zhou Quoc V. Le Barret Zoph Jason W. Wei Adam Roberts ALM 77 660 0 31 Jan 2023
Prediction-Powered Inference Anastasios Nikolas Angelopoulos Stephen Bates Clara Fannjiang Michael I. Jordan Tijana Zrnic 120 95 0 23 Jan 2023
Large Language Models Encode Clinical Knowledge K. Singhal Shekoofeh Azizi T. Tu S. S. Mahdavi Jason W. Wei ... A. Rajkomar Joelle Barral Christopher Semturs Alan Karthikesalingam Vivek Natarajan LM&MA ELM AI4MH 106 2,283 0 26 Dec 2022
Self-Instruct: Aligning Language Models with Self-Generated Instructions Yizhong Wang Yeganeh Kordi Swaroop Mishra Alisa Liu Noah A. Smith Daniel Khashabi Hannaneh Hajishirzi ALM SyDa LRM 79 2,166 0 20 Dec 2022
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming Hussein Mozannar Gagan Bansal Adam Fourney Eric Horvitz 78 110 0 25 Oct 2022
On the Limitations of Elo: Real-World Games, are Transitive, not Additive Quentin Bertrand Wojciech M. Czarnecki Gauthier Gidel 47 22 0 21 Jun 2022
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen ... Jack Clark Sam McCandlish C. Olah Benjamin Mann Jared Kaplan 212 2,457 0 12 Apr 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 694 12,525 0 04 Mar 2022
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts Stephen H. Bach Victor Sanh Zheng-Xin Yong Albert Webson Colin Raffel ... Khalid Almubarak Xiangru Tang Dragomir R. Radev Mike Tian-Jian Jiang Alexander M. Rush VLM 301 345 0 02 Feb 2022
A General Language Assistant as a Laboratory for Alignment Amanda Askell Yuntao Bai Anna Chen Dawn Drain Deep Ganguli ... Tom B. Brown Jack Clark Sam McCandlish C. Olah Jared Kaplan ALM 94 762 0 01 Dec 2021
Finetuned Language Models Are Zero-Shot Learners Jason W. Wei Maarten Bosma Vincent Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai Quoc V. Le ALM UQCV 64 3,678 0 03 Sep 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 155 5,328 0 07 Jul 2021
Cross-Task Generalization via Natural Language Crowdsourcing Instructions Swaroop Mishra Daniel Khashabi Chitta Baral Hannaneh Hajishirzi LRM 106 732 0 18 Apr 2021
Measuring Massive Multitask Language Understanding Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika D. Song Jacob Steinhardt ELM RALM 137 4,222 0 07 Sep 2020
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge Alon Talmor Jonathan Herzig Nicholas Lourie Jonathan Berant RALM 115 1,677 0 02 Nov 2018
Using the bootstrap to quantify the authority of an empirical ranking P. Hall Hugh Miller 100 51 0 19 Nov 2009