Evaluation of Retrieval-Augmented Generation: A Survey

13 May 2024

Kai Zhang

Qi Liu

Papers citing "Evaluation of Retrieval-Augmented Generation: A Survey"

50 / 61 papers shown

Title
Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets Lorenz Brehme Thomas Ströhle Ruth Breu 178 0 0 28 Apr 2025
Affordable AI Assistants with Knowledge Graph of Thoughts Maciej Besta Lorenzo Paleari Jia Hao Andrea Jiang Robert Gerstenberger You Wu ... Torsten Hoefler Grzegorz Kwa'sniewski Marcin Copik H. Niewiadomski Torsten Hoefler LLMAG RALM 459 0 0 03 Apr 2025
Reference-Aligned Retrieval-Augmented Question Answering over Heterogeneous Proprietary Documents Nayoung Choi Grace Byun Andrew Chung Ellie S. Paek S. Lee Jinho D. Choi RALM 212 1 0 26 Feb 2025
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation Satyapriya Krishna Kalpesh Krishna Anhad Mohananey Steven Schwarcz Adam Stambler Shyam Upadhyay Manaal Faruqui ReLM 3DV LRM RALM 69 28 0 28 Jan 2025
ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems Mohita Chowdhury Yajie Vera He Aisling Higham Ernest Lim 141 1 0 14 Jan 2025
GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering Sacha Muller António Loison Bilel Omrani Gautier Viaud RALM ELM 72 2 0 10 Sep 2024
DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation Shuting Wang Jiongnan Liu Shiren Song Jiehan Cheng Yuqi Fu Peidong Guo Kun Fang Yutao Zhu Zhicheng Dou RALM 66 10 0 09 Jun 2024
A Survey on Retrieval-Augmented Text Generation for Large Language Models Yizheng Huang Jimmy X. Huang 3DV RALM 112 50 0 17 Apr 2024
Retrieval-Augmented Generation for AI-Generated Content: A Survey Penghao Zhao Hailin Zhang Qinhan Yu Zhengren Wang Yunteng Geng Fangcheng Fu Ling Yang Wentao Zhang Jie Jiang Tengjiao Wang 3DV 212 273 0 29 Feb 2024
Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark Zhikun Xu Hai-Tao Zheng Ruixue Ding Xinyu Wang Boli Chen Yong Jiang Hai-Tao Zheng Wenlian Lu Pengjun Xie Fei Huang 88 11 0 29 Feb 2024
Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents Corby Rosset Ho-Lam Chung Guanghui Qin Ethan C. Chau Zhuo Feng Ahmed Hassan Awadallah Jennifer Neville Nikhil Rao 57 13 0 27 Feb 2024
Benchmarking Retrieval-Augmented Generation for Medicine Guangzhi Xiong Qiao Jin Zhiyong Lu Aidong Zhang RALM 123 186 0 20 Feb 2024
FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation Shuai Wang Ekaterina Khramtsova Shengyao Zhuang Guido Zuccon 81 13 0 19 Feb 2024
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models Yuanjie Lyu Zhiyu Li Pengnian Qi Feiyu Xiong Simin Niu Wenjin Wang Hao Wu Huan Liu Tong Xu Enhong Chen RALM 77 38 0 30 Jan 2024
MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries Yixuan Tang Yi Yang RALM 98 97 0 27 Jan 2024
The Power of Noise: Redefining Retrieval for RAG Systems Florin Cuconasu Giovanni Trappolini F. Siciliano Simone Filice Cesare Campagnano Y. Maarek Nicola Tonellotto Fabrizio Silvestri RALM 102 173 0 26 Jan 2024
RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture M. A. D. L. Balaguer Vinamra Benara Renato Luiz de Freitas Cunha Roberto de M. Estevao Filho Todd Hendry ... Morris Sharp B. Silva Swati Sharma Vijay Aski Ranveer Chandra FaML 89 88 0 16 Jan 2024
Seven Failure Points When Engineering a Retrieval Augmented Generation System Scott Barnett Stefanus Kurniawan Srikanth Thudumu Zach Brannelly Mohamed Abdelrazek 71 91 0 11 Jan 2024
Retrieval-Augmented Generation for Large Language Models: A Survey Yunfan Gao Yun Xiong Xinyu Gao Kangxiang Jia Jinliu Pan Yuxi Bi Yi Dai Jiawei Sun Meng Wang Haofen Wang 3DV RALM 155 1,776 1 18 Dec 2023
UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation Xun Liang Shichao Song Pengnian Qi Zhiyu Li Feiyu Xiong ... Zhaohui Wy Dawei He Peng Cheng Zhonghao Wang Haiying Deng HILM 63 22 0 26 Nov 2023
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems Jon Saad-Falcon Omar Khattab Christopher Potts Matei A. Zaharia RALM 75 115 0 16 Nov 2023
RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge Yi Liu Lianzhe Huang Shicheng Li Sishuo Chen Hao Zhou Fandong Meng Jie Zhou Xu Sun RALM 99 35 0 14 Nov 2023
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions Lei Huang Weijiang Yu Weitao Ma Weihong Zhong Zhangyin Feng ... Qianglong Chen Weihua Peng Xiaocheng Feng Bing Qin Ting Liu LRM HILM 126 868 0 09 Nov 2023
Evaluating Generative Ad Hoc Information Retrieval Lukas Gienapp Harrisen Scells Niklas Deckers Janek Bevendorff Shuai Wang ... Maik Fröbe Guide Zucoon Benno Stein Matthias Hagen Martin Potthast RALM 98 13 0 08 Nov 2023
Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves Yihe Deng Weitong Zhang Zixiang Chen Quanquan Gu LRM 94 83 0 07 Nov 2023
On the Opportunities of Green Computing: A Survey You Zhou Xiujing Lin Xiang Zhang Maolin Wang Gangwei Jiang ... Chao Mou Shuai Han Wuxia Jin Guannan Zhang Xiaodong Zeng 40 10 0 01 Nov 2023
Merging Generated and Retrieved Knowledge for Open-Domain QA Yunxiang Zhang Muhammad Khalifa Lajanugen Logeswaran Moontae Lee Honglak Lee Lu Wang RALM 69 38 0 22 Oct 2023
ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks Xiaodong Yu Hao Cheng Xiaodong Liu Dan Roth Jianfeng Gao HILM AAML 48 16 0 19 Oct 2023
LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples Jia-Yu Yao Kun-Peng Ning Zhen-Hui Liu Munan Ning Li Yuan HILM LRM AAML 63 190 0 02 Oct 2023
Ragas: Automated Evaluation of Retrieval Augmented Generation ES Shahul Jithin James Luis Espinosa-Anke Steven Schockaert 120 190 0 26 Sep 2023
Benchmarking Large Language Models in Retrieval-Augmented Generation Jiawei Chen Hongyu Lin Xianpei Han Le Sun 3DV RALM 62 296 0 04 Sep 2023
Graph of Thoughts: Solving Elaborate Problems with Large Language Models Maciej Besta Nils Blach Aleš Kubíček Robert Gerstenberger Michal Podstawski ... Joanna Gajda Tomasz Lehmann H. Niewiadomski Piotr Nyczyk Torsten Hoefler LRM AI4CE LM&Ro 131 671 0 18 Aug 2023
A Survey on Evaluation of Large Language Models Yu-Chu Chang Xu Wang Jindong Wang Yuanyi Wu Linyi Yang ... Yue Zhang Yi-Ju Chang Philip S. Yu Qian Yang Xingxu Xie ELM LM&MA ALM 120 1,686 0 06 Jul 2023
Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard Ehsan Kamalloo Nandan Thakur Carlos Lassance Xueguang Ma Jheng-Hong Yang Jimmy J. Lin 67 15 0 13 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 354 4,312 0 09 Jun 2023
Tree of Thoughts: Deliberate Problem Solving with Large Language Models Shunyu Yao Dian Yu Jeffrey Zhao Izhak Shafran Thomas Griffiths Yuan Cao Karthik Narasimhan LM&Ro LRM AI4CE 138 1,949 0 17 May 2023
Learning to Reason and Memorize with Self-Notes Jack Lanchantin Shubham Toshniwal Jason Weston Arthur Szlam Sainbayar Sukhbaatar ReLM LRM LLMAG 119 29 0 01 May 2023
A Survey for Efficient Open Domain Question Answering Qin Zhang Shan Chen Dongkuan Xu Qingqing Cao Xiaojun Chen Trevor Cohn Meng Fang 62 34 0 15 Nov 2022
FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation Sebastian Hofstatter Jiecao Chen K. Raman Hamed Zamani RALM 84 82 0 28 Sep 2022
RealTime QA: What's the Answer Right Now? Jungo Kasai Keisuke Sakaguchi Yoichi Takahashi Ronan Le Bras Akari Asai Xinyan Velocity Yu Dragomir R. Radev Noah A. Smith Yejin Choi Kentaro Inui KELM 130 192 0 27 Jul 2022
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 279 2,480 0 15 Jun 2022
Incorporating Dynamic Semantics into Pre-Trained Language Model for Aspect-based Sentiment Analysis Kai Zhang Kunpeng Zhang Mengdi Zhang Hongke Zhao Qi Liu Wei Wu Enhong Chen 31 52 0 30 Mar 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 874 12,973 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 814 9,387 0 28 Jan 2022
Understanding Jargon: Combining Extraction and Generation for Definition Modeling Jie Huang Hanyin Shao Kevin Chen-Chuan Chang Jinjun Xiong Wen-mei W. Hwu 38 17 0 14 Nov 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 466 10,367 0 17 Jun 2021
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering Fengbin Zhu Wenqiang Lei Chao Wang Jianming Zheng Soujanya Poria Tat-Seng Chua RALM 247 256 0 04 Jan 2021
A Survey of Evaluation Metrics Used for NLG Systems Ananya B. Sai Akash Kumar Mohankumar Mitesh M. Khapra ELM 87 236 0 27 Aug 2020
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT Omar Khattab Matei A. Zaharia 138 1,370 0 27 Apr 2020
ERASER: A Benchmark to Evaluate Rationalized NLP Models Jay DeYoung Sarthak Jain Nazneen Rajani Eric P. Lehman Caiming Xiong R. Socher Byron C. Wallace 112 637 0 08 Nov 2019