Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

2 June 2023

Weijia Shi

Prithviraj Ammanabrolu

Mari Ostendorf

Papers citing "Fine-Grained Human Feedback Gives Better Rewards for Language Model Training"

50 / 254 papers shown

Title
HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM Zhilin Wang Yi Dong Jiaqi Zeng Virginia Adams Makesh Narsimhan Sreedhar ... Olivier Delalleau Jane Polak Scowcroft Neel Kant Aidan Swope Oleksii Kuchaiev 3DV 14 66 0 16 Nov 2023
Predicting Text Preference Via Structured Comparative Reasoning Jing Nathan Yan Tianqi Liu Justin T Chiu Jiaming Shen Zhen Qin ... Charumathi Lakshmanan Y. Kurzion Alexander M. Rush Jialu Liu Michael Bendersky LRM 48 7 0 14 Nov 2023
Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision Seongyun Lee Sue Hyun Park Yongrae Jo Minjoon Seo 30 50 0 13 Nov 2023
Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering Yichi Zhang Zhuo Chen Yin Fang Yanxi Lu Fangming Li Wen Zhang Hua-zeng Chen 66 30 0 11 Nov 2023
Let's Reinforce Step by Step Sarah Pan Vladislav Lialin Sherin Muckatira Anna Rumshisky ReLM LRM 14 7 0 10 Nov 2023
Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism Lang Cao 26 13 0 02 Nov 2023
Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions Taehyeon Kim Joonkee Kim Gihun Lee Se-Young Yun 33 11 0 01 Nov 2023
The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback Nathan Lambert Roberto Calandra ALM 29 32 0 31 Oct 2023
Safe RLHF: Safe Reinforcement Learning from Human Feedback Josef Dai Xuehai Pan Ruiyang Sun Jiaming Ji Xinbo Xu Mickel Liu Yizhou Wang Yaodong Yang 38 293 0 19 Oct 2023
Know Where to Go: Make LLM a Relevant, Responsible, and Trustworthy Searcher Xiang Shi Jiawei Liu Yinpeng Liu Qikai Cheng Wei Lu RALM HILM KELM 29 6 0 19 Oct 2023
MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models Deepak Nathani David Wang Liangming Pan Wei Wang KELM LRM ReLM 28 10 0 19 Oct 2023
Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging Joel Jang Seungone Kim Bill Yuchen Lin Yizhong Wang Jack Hessel Luke Zettlemoyer Hannaneh Hajishirzi Yejin Choi Prithviraj Ammanabrolu MoMe 57 133 0 17 Oct 2023
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection Akari Asai Zeqiu Wu Yizhong Wang Avirup Sil Hannaneh Hajishirzi RALM 176 638 0 17 Oct 2023
Let's reward step by step: Step-Level reward model as the Navigators for Reasoning Qianli Ma Haotian Zhou Tingkai Liu Jianbo Yuan Pengfei Liu Yang You Hongxia Yang LRM 35 43 0 16 Oct 2023
Evaluating Large Language Models at Evaluating Instruction Following Zhiyuan Zeng Jiatong Yu Tianyu Gao Yu Meng Tanya Goyal Danqi Chen ELM ALM 46 167 0 11 Oct 2023
The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values Hannah Rose Kirk Andrew M. Bean Bertie Vidgen Paul Röttger Scott A. Hale ALM 21 42 0 11 Oct 2023
Constructive Large Language Models Alignment with Diverse Feedback Tianshu Yu Ting-En Lin Yuchuan Wu Min Yang Fei Huang Yongbin Li ALM 40 9 0 10 Oct 2023
Crystal: Introspective Reasoners Reinforced with Self-Feedback Jiacheng Liu Ramakanth Pasunuru Hannaneh Hajishirzi Yejin Choi Asli Celikyilmaz LRM ReLM 31 22 0 07 Oct 2023
Confronting Reward Model Overoptimization with Constrained RLHF Ted Moskovitz Aaditya K. Singh DJ Strouse T. Sandholm Ruslan Salakhutdinov Anca D. Dragan Stephen Marcus McAleer 39 48 0 06 Oct 2023
A Long Way to Go: Investigating Length Correlations in RLHF Prasann Singhal Tanya Goyal Jiacheng Xu Greg Durrett 44 143 0 05 Oct 2023
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization Zhanhui Zhou Jie Liu Chao Yang Jing Shao Yu Liu Xiangyu Yue Wanli Ouyang Yu Qiao 40 49 0 05 Oct 2023
The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Models Hannah Rose Kirk Bertie Vidgen Paul Röttger Scott A. Hale 47 2 0 03 Oct 2023
Human Feedback is not Gold Standard Tom Hosking Phil Blunsom Max Bartolo ALM 32 49 0 28 Sep 2023
The Trickle-down Impact of Reward (In-)consistency on RLHF Lingfeng Shen Sihao Chen Linfeng Song Lifeng Jin Baolin Peng Haitao Mi Daniel Khashabi Dong Yu 40 21 0 28 Sep 2023
Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding Jiacheng Liu Andrew Cohen Ramakanth Pasunuru Yejin Choi Hannaneh Hajishirzi Asli Celikyilmaz 24 24 0 26 Sep 2023
Chain-of-Verification Reduces Hallucination in Large Language Models S. Dhuliawala M. Komeili Jing Xu Roberta Raileanu Xian Li Asli Celikyilmaz Jason Weston LRM HILM 22 177 0 20 Sep 2023
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data Guan-Bo Wang Sijie Cheng Xianyuan Zhan Xiangang Li Sen Song Yang Liu ALM 27 231 0 20 Sep 2023
Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF Simeng Sun Dhawal Gupta Mohit Iyyer 27 17 0 16 Sep 2023
$Decolonial AI Alignment: Openness, Viśe\d{s}a-Dharma, and Including Excluded Knowledges$ Decolonial AI Alignment: Openness, Viśe\d{s}a-Dharma, and Including Excluded Knowledges Kush R. Varshney 44 2 0 10 Sep 2023
Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties Taylor Sorensen Liwei Jiang Jena D. Hwang Sydney Levine Valentina Pyatkin ... Kavel Rao Chandra Bhagavatula Maarten Sap J. Tasioulas Yejin Choi SLR 24 51 0 02 Sep 2023
Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models Hritik Bansal John Dang Aditya Grover ALM 35 20 0 30 Aug 2023
Reinforcement Learning for Generative AI: A Survey Yuanjiang Cao Quan.Z Sheng Julian McAuley Lina Yao SyDa 48 10 0 28 Aug 2023
Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operation Xinshuo Hu Dongfang Li Baotian Hu Zihao Zheng Zhenyu Liu Hao Fei KELM MU 35 26 0 16 Aug 2023
The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation Patrick Fernandes Daniel Deutsch M. Finkelstein Parker Riley André F. T. Martins Graham Neubig Ankush Garg J. Clark Markus Freitag Orhan Firat LRM 36 68 0 14 Aug 2023
Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation David Heineman Yao Dou Wei-ping Xu 32 7 0 14 Aug 2023
Detecting and Preventing Hallucinations in Large Vision Language Models Anisha Gunjal Jihan Yin Erhan Bas MLLM VLM 36 156 0 11 Aug 2023
ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIF Víctor Gallego SyDa 51 4 0 11 Aug 2023
Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies Liangming Pan Michael Stephen Saxon Wenda Xu Deepak Nathani Xinyi Wang William Yang Wang KELM LRM 47 201 0 06 Aug 2023
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback Stephen Casper Xander Davies Claudia Shi T. Gilbert Jérémy Scheurer ... Erdem Biyik Anca Dragan David M. Krueger Dorsa Sadigh Dylan Hadfield-Menell ALM OffRL 52 473 0 27 Jul 2023
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets Seonghyeon Ye Doyoung Kim Sungdong Kim Hyeonbin Hwang Seungone Kim Yongrae Jo James Thorne Juho Kim Minjoon Seo ALM 46 99 0 20 Jul 2023
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset Jiaming Ji Mickel Liu Juntao Dai Xuehai Pan Chi Zhang Ce Bian Chi Zhang Ruiyang Sun Yizhou Wang Yaodong Yang ALM 25 411 0 10 Jul 2023
Let Me Teach You: Pedagogical Foundations of Feedback for Language Models Beatriz Borges Niket Tandon Tanja Kaser Antoine Bosselut 22 4 0 01 Jul 2023
Preference Ranking Optimization for Human Alignment Feifan Song Yu Bowen Minghao Li Haiyang Yu Fei Huang Yongbin Li Houfeng Wang ALM 26 238 0 30 Jun 2023
AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks Alexander Tornede Difan Deng Theresa Eimer Joseph Giovanelli Aditya Mohan ... Sarah Segel Daphne Theodorakopoulos Tanja Tornede Henning Wachsmuth Marius Lindauer 36 23 0 13 Jun 2023
Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards Alexandre Ramé Guillaume Couairon Mustafa Shukor Corentin Dancette Jean-Baptiste Gaya Laure Soulier Matthieu Cord MoMe 35 136 0 07 Jun 2023
Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models Ashutosh Baheti Ximing Lu Faeze Brahman Ronan Le Bras Maarten Sap Mark O. Riedl 38 9 0 24 May 2023
DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4 Ye Hu Kaiqiang Song Sangwoo Cho Xiaoyang Wang H. Foroosh Fei Liu 31 11 0 24 May 2023
Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA David Heineman Yao Dou Mounica Maddela Wei-ping Xu 23 15 0 23 May 2023
LLM-powered Data Augmentation for Enhanced Cross-lingual Performance Chenxi Whitehouse Monojit Choudhury Alham Fikri Aji SyDa LRM 32 68 0 23 May 2023
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels Weiyan Shi Emily Dinan Kurt Shuster Jason Weston Jing Xu 52 19 0 28 Oct 2022