FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets

20 July 2023

Papers citing "FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets"

50 / 91 papers shown

Title
"There Is No Such Thing as a Dumb Question," But There Are Good Ones Minjung Shin Donghyun Kim Jeh-Kwang Ryu ELM 28 0 0 15 May 2025
TRAIL: Trace Reasoning and Agentic Issue Localization Darshan Deshpande Varun Gangal Hersh Mehta Jitin Krishnan Anand Kannappan Rebecca Qian 25 0 0 13 May 2025
A Cost-Effective LLM-based Approach to Identify Wildlife Trafficking in Online Marketplaces Juliana Barbosa Ulhas Gondhali Gohar Petrossian Kinshuk Sharma Sunandan Chakraborty Jennifer Jacquet Juliana Freire 31 0 0 29 Apr 2025
Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation Tuhina Tripathi Manya Wadhwa Greg Durrett S. Niekum 32 0 0 20 Apr 2025
FinGrAct: A Framework for FINe-GRrained Evaluation of ACTionability in Explainable Automatic Fact-Checking Islam Eldifrawi Shengrui Wang Amine Trabelsi 29 0 0 07 Apr 2025
Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models José P. Pombal Nuno M. Guerreiro Ricardo Rei André F. T. Martins ALM 75 0 0 01 Apr 2025
KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language Yoonshik Kim Jaeyoon Jung 37 0 0 31 Mar 2025
ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedback Taewon Yun Jihwan Oh Hyangsuk Min Yuho Lee Jihwan Bang Jason (Jinglun) Cai Hwanjun Song OffRL LRM 39 0 0 27 Mar 2025
Scaling Evaluation-time Compute with Reasoning Models as Process Evaluators Seungone Kim Ian Wu Jinu Lee Xiang Yue Seongyun Lee ... Kiril Gashteovski Carolin (Haas) Lawrence J. Hockenmaier Graham Neubig Sean Welleck LRM 50 2 0 25 Mar 2025
A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications Jian-Yu Guan Jian Wu J. Li Chuanqi Cheng Wei Yu Wu LM&MA 74 0 0 21 Mar 2025
REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities Alexander Pugachev Alena Fenogenova Vladislav Mikhailov Ekaterina Artemova 37 0 0 17 Mar 2025
Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performance Bryan Etzine Masoud Hashemi Nishanth Madhusudhan Sagar Davasam Roshnee Sharma Sathwik Tejaswi Madhusudhan Vikas Yadav 39 0 0 07 Mar 2025
Improving LLM-as-a-Judge Inference with the Judgment Distribution Victor Wang Michael J.Q. Zhang Eunsol Choi 55 0 0 04 Mar 2025
What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text Arturs Kanepajs Aditi Basu Sankalpa Ghose Constance Li Akshat Mehta Ronak Mehta Samuel David Tucker-Davis Eric Zhou Bob Fischer ALM ELM 43 0 0 03 Mar 2025
BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge Terry Tong Fei-Yue Wang Zhe Zhao M. Chen AAML ELM 37 1 0 01 Mar 2025
System Message Generation for User Preferences using Open-Source Models Minbyul Jeong Jungho Cho Minsoo Khang Dawoon Jung Teakgyu Hong 41 0 0 17 Feb 2025
Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation Takyoung Kim Kyungjae Lee Y. Jang Ji Yong Cho Gangwoo Kim Minseok Cho Moontae Lee 154 0 0 28 Jan 2025
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking Benjamin Feuer Micah Goldblum Teresa Datta Sanjana Nambiar Raz Besaleli Samuel Dooley Max Cembalest John P. Dickerson ALM 39 7 0 28 Jan 2025
RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems Robert Friel Masha Belyi Atindriyo Sanyal 82 19 0 17 Jan 2025
Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment Jianfei Zhang Jun Bai Yangqiu Song Yanmeng Wang Rumei Li Chenghua Lin Wenge Rong 44 0 0 31 Dec 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 120 67 0 25 Nov 2024
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following Gabrielle Kaili-May Liu Bowen Shi Avi Caciularu Idan Szpektor Arman Cohan 72 4 0 30 Oct 2024
Uncovering Factor Level Preferences to Improve Human-Model Alignment Juhyun Oh Eunsu Kim Jiseon Kim Wenda Xu Inha Cha William Yang Wang Alice H. Oh 34 0 0 09 Oct 2024
Better Instruction-Following Through Minimum Bayes Risk Ian Wu Patrick Fernandes Amanda Bertsch Seungone Kim Sina Pakazad Graham Neubig 48 9 0 03 Oct 2024
Beyond Scalar Reward Model: Learning Generative Judge from Preference Data Ziyi Ye Xiangsheng Li Qiuchi Li Qingyao Ai Yujia Zhou Wei Shen Dong Yan Yiqun Liu 50 10 0 01 Oct 2024
Direct Judgement Preference Optimization Peifeng Wang Austin Xu Yilun Zhou Caiming Xiong Shafiq Joty ELM 39 12 0 23 Sep 2024
Beyond Persuasion: Towards Conversational Recommender System with Credible Explanations Peixin Qin Chen Huang Yang Deng Wenqiang Lei Tat-Seng Chua LRM 27 3 0 22 Sep 2024
Aligning Language Models Using Follow-up Likelihood as Reward Signal Chen Zhang Dading Chong Feng Jiang Chengguang Tang Anningzhe Gao Guohua Tang Haizhou Li ALM 33 2 0 20 Sep 2024
LLM-as-a-Judge & Reward Model: What They Can and Cannot Do Guijin Son Hyunwoo Ko Hoyoung Lee Yewon Kim Seunghyeok Hong ALM ELM 51 6 0 17 Sep 2024
Your Weak LLM is Secretly a Strong Teacher for Alignment Leitian Tao Yixuan Li 88 5 0 13 Sep 2024
Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions Bhuvanashree Murugadoss Christian Poelitz Ian Drosos Vu Le Nick McKenna Carina Negreanu Chris Parnin Advait Sarkar ELM ALM 35 13 0 16 Aug 2024
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning Prateek Yadav Colin Raffel Mohammed Muqeeth Lucas Caccia Haokun Liu Tianlong Chen Joey Tianyi Zhou Leshem Choshen Alessandro Sordoni MoMe 46 21 0 13 Aug 2024
Self-Taught Evaluators Tianlu Wang Ilia Kulikov O. Yu. Golovneva Ping Yu Weizhe Yuan Jane Dwivedi-Yu Richard Yuanzhe Pang Maryam Fazel-Zarandi Jason Weston Xian Li ALM LRM 29 22 0 05 Aug 2024
Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Education Seungyoon Kim Seungone Kim AI4Ed 31 0 0 24 Jul 2024
Improving Context-Aware Preference Modeling for Language Models Silviu Pitis Ziang Xiao Nicolas Le Roux Alessandro Sordoni 38 8 0 20 Jul 2024
How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studies Alina Leidinger Richard Rogers 34 5 0 16 Jul 2024
STORYSUMM: Evaluating Faithfulness in Story Summarization Melanie Subbiah Faisal Ladhak Akankshya Mishra Griffin Adams Lydia B. Chilton Kathleen McKeown 48 4 0 09 Jul 2024
Diverse and Fine-Grained Instruction-Following Ability Exploration with Synthetic Data Zihui Gu Xingwu Sun Fengzong Lian Zhanhui Kang Cheng-zhong Xu Ju Fan ALM 40 1 0 04 Jul 2024
Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment Janghwan Lee Seongmin Park S. Hong Minsoo Kim Du-Seong Chang Jungwook Choi 29 4 0 03 Jul 2024
Model Merging and Safety Alignment: One Bad Model Spoils the Bunch Hasan Hammoud Umberto Michieli Fabio Pizzati Philip H. S. Torr Adel Bibi Guohao Li Mete Ozay MoMe 31 15 0 20 Jun 2024
Finding Blind Spots in Evaluator LLMs with Interpretable Checklists Sumanth Doddapaneni Mohammed Safi Ur Rahman Khan Sshubam Verma Mitesh Khapra 39 11 0 19 Jun 2024
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models Seungone Kim Juyoung Suk Ji Yong Cho Shayne Longpre Chaeeun Kim ... Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo ELM ALM LM&MA 99 31 0 09 Jun 2024
Mixture-of-Agents Enhances Large Language Model Capabilities Junlin Wang Jue Wang Ben Athiwaratkun Ce Zhang James Zou LLMAG AIFin 41 99 0 07 Jun 2024
TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models Chen Zhang Chengguang Tang Dading Chong Ke Shi Guohua Tang Feng Jiang Haizhou Li 35 4 0 30 May 2024
Aligning to Thousands of Preferences via System Message Generalization Seongyun Lee Sue Hyun Park Seungone Kim Minjoon Seo ALM 41 38 0 28 May 2024
HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants Milan Gritta Gerasimos Lampouras Ignacio Iacobacci ALM 29 1 0 15 May 2024
PHUDGE: Phi-3 as Scalable Judge Mahesh Deshwal Apoorva Chawla ALM 19 0 0 12 May 2024
R4: Reinforced Retriever-Reorder-Responder for Retrieval-Augmented Large Language Models Taolin Zhang Dongyang Li Qizhou Chen Chengyu Wang Longtao Huang Hui Xue Xiaofeng He Junyuan Huang RALM KELM 50 0 0 04 May 2024
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models Seungone Kim Juyoung Suk Shayne Longpre Bill Yuchen Lin Jamin Shin Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo MoMe ALM ELM 51 168 0 02 May 2024
FLAME: Factuality-Aware Alignment for Large Language Models Sheng-Chieh Lin Luyu Gao Barlas Oğuz Wenhan Xiong Jimmy Lin Wen-tau Yih Xilun Chen HILM 38 14 0 02 May 2024