Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

2 June 2023

Weijia Shi

Prithviraj Ammanabrolu

Mari Ostendorf

Papers citing "Fine-Grained Human Feedback Gives Better Rewards for Language Model Training"

50 / 254 papers shown

Title
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs Shreyas Chaudhari Pranjal Aggarwal Vishvak Murahari Tanmay Rajpurohit Ashwin Kalyan Karthik R. Narasimhan Ameet Deshpande Bruno Castro da Silva 29 34 0 12 Apr 2024
FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback Liqiang Jing Xinya Du 79 17 0 07 Apr 2024
ROPO: Robust Preference Optimization for Large Language Models Xize Liang Chao Chen Shuang Qiu Jie Wang Yue-bo Wu Zhihang Fu Zhihao Shi Feng Wu Jieping Ye 48 1 0 05 Apr 2024
Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data Jingyu Zhang Marc Marone Tianjian Li Benjamin Van Durme Daniel Khashabi 93 9 0 05 Apr 2024
Hallucination Diversity-Aware Active Learning for Text Summarization Yu Xia Xu Liu Tong Yu Sungchul Kim Ryan A. Rossi Anup B. Rao Tung Mai Shuai Li HILM 40 3 0 02 Apr 2024
Prior Constraints-based Reward Model Training for Aligning Large Language Models Hang Zhou Chenglong Wang Yimin Hu Tong Xiao Chunliang Zhang Jingbo Zhu ALM 46 2 0 01 Apr 2024
Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization Hritik Bansal Ashima Suvarna Gantavya Bhatt Nanyun Peng Kai-Wei Chang Aditya Grover ALM 64 9 0 31 Mar 2024
MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models Kailai Yang Zhiwei Liu Qianqian Xie Jimin Huang Tianlin Zhang Sophia Ananiadou 34 15 0 25 Mar 2024
Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Models Minchan Kim Minyeong Kim Junik Bae Suhwan Choi Sungkyung Kim Buru Chang VLM 32 3 0 24 Mar 2024
Opportunities and challenges in the application of large artificial intelligence models in radiology Liangrui Pan Zhenyu Zhao Ying Lu Kewei Tang Liyong Fu Qingchun Liang Shaoliang Peng LM&MA MedIm AI4CE 45 5 0 24 Mar 2024
Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection Kyungjae Lee Dasol Hwang Sunghyun Park Youngsoo Jang Moontae Lee 46 8 0 21 Mar 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James V. Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 85 218 0 20 Mar 2024
Reinforcement Learning with Token-level Feedback for Controllable Text Generation Wendi Li Wei Wei Kaihe Xu Wenfeng Xie Dangyang Chen Yu Cheng 41 7 0 18 Mar 2024
m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks Zixian Ma Weikai Huang Jieyu Zhang Tanmay Gupta Ranjay Krishna 55 18 0 17 Mar 2024
ALaRM: Align Language Models via Hierarchical Rewards Modeling Yuhang Lai Siyuan Wang Shujun Liu Xuanjing Huang Zhongyu Wei 31 4 0 11 Mar 2024
Calibrating Large Language Models Using Their Generations Only Dennis Ulmer Martin Gubri Hwaran Lee Sangdoo Yun Seong Joon Oh UQLM 432 18 1 09 Mar 2024
DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation Xueqing Wu Rui Zheng Jingzhen Sha Te-Lin Wu Hanyu Zhou Mohan Tang Kai-Wei Chang Nanyun Peng Haoran Huang 55 2 0 04 Mar 2024
Enhancing LLM Safety via Constrained Direct Preference Optimization Zixuan Liu Xiaolin Sun Zizhan Zheng 41 20 0 04 Mar 2024
Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment Yiju Guo Ganqu Cui Lifan Yuan Ning Ding Jiexin Wang ... Ruobing Xie Jie Zhou Yankai Lin Zhiyuan Liu Maosong Sun 36 60 0 29 Feb 2024
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards Haoxiang Wang Yong Lin Wei Xiong Rui Yang Shizhe Diao Shuang Qiu Han Zhao Tong Zhang 40 72 0 28 Feb 2024
Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step Li Zhong Zilong Wang Jingbo Shang 29 48 0 25 Feb 2024
Brain-Inspired Two-Stage Approach: Enhancing Mathematical Reasoning by Imitating Human Thought Processes Yezeng Chen Zui Chen Yi Zhou LRM 38 2 0 23 Feb 2024
Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models Loka Li Zhenhao Chen Guan-Hong Chen Yixuan Zhang Yusheng Su Eric P. Xing Kun Zhang LRM 44 16 0 19 Feb 2024
AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition Zhaorun Chen Zhuokai Zhao Zhihong Zhu Ruiqi Zhang Xiang Li Bhiksha Raj Huaxiu Yao LRM 30 25 0 18 Feb 2024
Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment Rui Yang Xiaoman Pan Feng Luo Shuang Qiu Han Zhong Dong Yu Jianshu Chen 103 69 0 15 Feb 2024
Reward Generalization in RLHF: A Topological Perspective Tianyi Qiu Fanzhi Zeng Jiaming Ji Dong Yan Kaile Wang Jiayi Zhou Yang Han Josef Dai Xuehai Pan Yaodong Yang AI4CE 32 3 0 15 Feb 2024
Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey Zhichen Dong Zhanhui Zhou Chao Yang Jing Shao Yu Qiao ELM 52 58 0 14 Feb 2024
ODIN: Disentangled Reward Mitigates Hacking in RLHF Lichang Chen Chen Zhu Davit Soselia Jiuhai Chen Dinesh Manocha Tom Goldstein Heng-Chiao Huang M. Shoeybi Bryan Catanzaro AAML 50 53 0 11 Feb 2024
Training Language Models to Generate Text with Citations via Fine-grained Rewards Chengyu Huang Zeqiu Wu Yushi Hu Wenya Wang HILM LRM 79 27 0 06 Feb 2024
Personalized Language Modeling from Personalized Human Feedback Xinyu Li Zachary C. Lipton Liu Leqi ALM 71 48 0 06 Feb 2024
Rethinking the Role of Proxy Rewards in Language Model Alignment Sungdong Kim Minjoon Seo SyDa ALM 31 0 0 02 Feb 2024
Dense Reward for Free in Reinforcement Learning from Human Feedback Alex J. Chan Hao Sun Samuel Holt M. Schaar 23 32 0 01 Feb 2024
Transforming and Combining Rewards for Aligning Large Language Models Zihao Wang Chirag Nagpal Jonathan Berant Jacob Eisenstein Alex DÁmour Oluwasanmi Koyejo Victor Veitch 21 11 0 01 Feb 2024
Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble Shun Zhang Zhenfang Chen Sunli Chen Yikang Shen Zhiqing Sun Chuang Gan 31 26 0 30 Jan 2024
Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models Minbyul Jeong Jiwoong Sohn Mujeen Sung Jaewoo Kang 23 29 0 27 Jan 2024
ARGS: Alignment as Reward-Guided Search Maxim Khanov Jirayu Burapacheep Yixuan Li 35 46 0 23 Jan 2024
WARM: On the Benefits of Weight Averaged Reward Models Alexandre Ramé Nino Vieillard Léonard Hussenot Robert Dadashi Geoffrey Cideron Olivier Bachem Johan Ferret 120 94 0 22 Jan 2024
Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation Meng Cao Lei Shu Lei Yu Yun Zhu Nevan Wichers Yinxiao Liu Lei Meng OffRL ALM 27 4 0 14 Jan 2024
The Critique of Critique Shichao Sun Junlong Li Weizhe Yuan Ruifeng Yuan Wenjie Li Pengfei Liu ELM 40 0 0 09 Jan 2024
Reasons to Reject? Aligning Language Models with Judgments Weiwen Xu Deng Cai Zhisong Zhang Wai Lam Shuming Shi ALM 21 14 0 22 Dec 2023
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations Peiyi Wang Lei Li Zhihong Shao R. X. Xu Damai Dai Yifei Li Deli Chen Y.Wu Zhifang Sui AIMat LRM ALM 53 277 0 14 Dec 2023
On Diversified Preferences of Large Language Model Alignment Dun Zeng Yong Dai Pengyu Cheng Longyue Wang Tianhao Hu Wanshun Chen Nan Du Zenglin Xu ALM 38 16 0 12 Dec 2023
ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference Tianchi Cai Xierui Song Jiyan Jiang Fei Teng Jinjie Gu Guannan Zhang ALM 21 4 0 05 Dec 2023
Axiomatic Preference Modeling for Longform Question Answering Corby Rosset Guoqing Zheng Victor C. Dibia Ahmed Hassan Awadallah Paul Bennett SyDa 27 3 0 02 Dec 2023
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback M. Steyvers Yuan Yao Haoye Zhang Taiwen He Yifeng Han ... Xinyue Hu Zhiyuan Liu Hai-Tao Zheng Maosong Sun Tat-Seng Chua MLLM VLM 150 178 0 01 Dec 2023
ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up? Hailin Chen Fangkai Jiao Xingxuan Li Chengwei Qin Mathieu Ravaut Ruochen Zhao Caiming Xiong Chenyu You ELM CLL AI4MH LRM ALM 85 27 0 28 Nov 2023
A Survey of the Evolution of Language Model-Based Dialogue Systems Hongru Wang Lingzhi Wang Yiming Du Liang Chen Jing Zhou Yufei Wang Kam-Fai Wong LRM 67 21 0 28 Nov 2023
Large Language Models Meet Computer Vision: A Brief Survey Raby Hamadi LM&MA 29 4 0 28 Nov 2023
Case Repositories: Towards Case-Based Reasoning for AI Alignment K. J. Kevin Feng Quan Ze Chen Inyoung Cheong King Xia Amy X. Zhang 30 10 0 18 Nov 2023
Effective Large Language Model Adaptation for Improved Grounding and Citation Generation Xi Ye Ruoxi Sun Sercan Ö. Arik Tomas Pfister HILM 34 25 0 16 Nov 2023