MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

3 June 2024

Graham Neubig

Yang You

ELM

ArXiv PDF HTML

Papers citing "MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures"

26 / 26 papers shown

Title
When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research Guijin Son Jiwoo Hong Honglu Fan Heejeong Nam Hyunwoo Ko ... Jinyeop Song Jinha Choi Gonçalo Paulo Youngjae Yu Stella Biderman 71 1 0 17 May 2025
Bielik 11B v2 Technical Report Krzysztof Ociepa Łukasz Flis Krzysztof Wróbel Adrian Gwoździej Remigiusz Kinas 59 0 0 05 May 2025
Bielik v3 Small: Technical Report Krzysztof Ociepa Łukasz Flis Remigiusz Kinas Krzysztof Wróbel Adrian Gwoździej 57 0 0 05 May 2025
Preference Leakage: A Contamination Problem in LLM-as-a-judge Dawei Li Renliang Sun Yue Huang Ming Zhong Bohan Jiang Jiawei Han Wei Wei Wei Wang Huan Liu 99 23 0 03 Feb 2025
RedPajama: an Open Dataset for Training Large Language Models Maurice Weber Daniel Y. Fu Quentin Anthony Yonatan Oren S. Adams ... Tri Dao Percy Liang Christopher Ré Irina Rish Ce Zhang 185 71 0 19 Nov 2024
Improving Model Evaluation using SMART Filtering of Benchmark Datasets Vipul Gupta Candace Ross David Pantoja R. Passonneau Megan Ung Adina Williams 216 2 0 26 Oct 2024
Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment Mingzhi Wang Chengdong Ma Qizhi Chen Linjian Meng Yang Han Jiancong Xiao Zhaowei Zhang Jing Huo Weijie Su Yaodong Yang 112 7 0 22 Oct 2024
MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems Nandan Thakur Suleman Kazi Ge Luo Jimmy J. Lin Amin Ahmad VLM RALM 118 7 0 17 Oct 2024
A Unified Approach to Routing and Cascading for LLMs Jasper Dekoninck Maximilian Baader Martin Vechev 96 2 0 14 Oct 2024
RMB: Comprehensively Benchmarking Reward Models in LLM Alignment Enyu Zhou Guodong Zheng Binghai Wang Zhiheng Xi Shihan Dou ... Yurong Mou Rui Zheng Tao Gui Qi Zhang Xuanjing Huang ALM 87 19 0 13 Oct 2024
FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback Yongbin Li Miao Zheng Fan Yang Bin Cui Tengjiao Wang Xin Wu Guosheng Dong Wentao Zhang ALM 78 7 0 12 Oct 2024
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation Jasper Dekoninck Maximilian Baader Martin Vechev ALM 132 0 0 01 Sep 2024
WildChat: 1M ChatGPT Interaction Logs in the Wild Wenting Zhao Xiang Ren Jack Hessel Claire Cardie Yejin Choi Yuntian Deng 67 208 0 02 May 2024
Learn Your Reference Model for Real Good Alignment Alexey Gorbatovski Boris Shaposhnikov Alexey Malakhov Nikita Surnachev Yaroslav Aksenov Ian Maksimov Nikita Balagansky Daniil Gavrilov OffRL 95 30 0 15 Apr 2024
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators Yann Dubois Balázs Galambosi Percy Liang Tatsunori Hashimoto ALM 81 359 0 06 Apr 2024
Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward Ruohong Zhang Liangke Gui Zhiqing Sun Yihao Feng Keyang Xu ... Di Fu Chunyuan Li Alexander G. Hauptmann Yonatan Bisk Yiming Yang MLLM 86 70 0 01 Apr 2024
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference Wei-Lin Chiang Lianmin Zheng Ying Sheng Anastasios Nikolas Angelopoulos Tianle Li ... Hao Zhang Banghua Zhu Michael I. Jordan Joseph E. Gonzalez Ion Stoica OSLM 102 536 0 07 Mar 2024
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes Lizhou Fan Wenyue Hua Lingyao Li Haoyang Ling Yongfeng Zhang LRM 51 52 0 22 Dec 2023
Rethinking Benchmark and Contamination for Language Models with Rephrased Samples Shuo Yang Wei-Lin Chiang Lianmin Zheng Joseph E. Gonzalez Ion Stoica ALM 48 122 0 08 Nov 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 300 4,186 0 09 Jun 2023
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback Yann Dubois Xuechen Li Rohan Taori Tianyi Zhang Ishaan Gulrajani Jimmy Ba Carlos Guestrin Percy Liang Tatsunori B. Hashimoto ALM 106 581 0 22 May 2023
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 140 1,893 0 16 Aug 2021
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks Nils Reimers Iryna Gurevych 841 11,979 0 27 Aug 2019
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 191 1,475 0 24 May 2019
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 183 2,610 0 09 May 2017
The Arcade Learning Environment: An Evaluation Platform for General Agents Marc G. Bellemare Yavar Naddaf J. Veness Michael Bowling 92 2,992 0 19 Jul 2012