Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

18 June 2024

Haoxiang Wang

Wei Xiong

Tengyang Xie

Han Zhao

Tong Zhang

ArXiv PDF HTML

Papers citing "Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts"

49 / 49 papers shown

Title
On the Robustness of Reward Models for Language Model Alignment Jiwoo Hong Noah Lee Eunki Kim Guijin Son Woojin Chung Aman Gupta Shao Tang James Thorne 29 0 0 12 May 2025
Sandcastles in the Storm: Revisiting the (Im)possibility of Strong Watermarking Fabrice Harel-Canada Boran Erol Connor Choi J. Liu Gary Jiarui Song Nanyun Peng Amit Sahai AAML 26 0 0 11 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 72 1 0 05 May 2025
RM-R1: Reward Modeling as Reasoning Xiusi Chen Gaotang Li Zehua Wang Bowen Jin Cheng Qian ... Y. Zhang D. Zhang Tong Zhang Hanghang Tong Heng Ji ReLM OffRL LRM 165 1 0 05 May 2025
Improving Model Alignment Through Collective Intelligence of Open-Source LLMS Junlin Wang Roy Xie Shang Zhu Jue Wang Ben Athiwaratkun Bhuwan Dhingra Shuaiwen Leon Song Ce Zhang James Zou ALM 31 0 0 05 May 2025
Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model Junshu Pan Wei Shen Shulin Huang Qiji Zhou Yue Zhang 71 0 0 22 Apr 2025
Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment Xiaotian Zhang Ruizhe Chen Yang Feng Zuozhu Liu 40 0 0 17 Apr 2025
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion Longguang Zhong Fanqi Wan Ziyi Yang Guosheng Liang Tianyuan Shi Xiaojun Quan MoMe 57 0 0 09 Apr 2025
Inference-Time Scaling for Generalist Reward Modeling Zijun Liu P. Wang Ran Xu Shirong Ma Chong Ruan Peng Li Yang Liu Y. Wu OffRL LRM 46 11 0 03 Apr 2025
Aligning Multimodal LLM with Human Preference: A Survey Tao Yu Yuyao Zhang Chaoyou Fu Junkang Wu Jinda Lu ... Qingsong Wen Z. Zhang Yan Huang Liang Wang Tieniu Tan 164 2 0 18 Mar 2025
PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing Cheng Deng Luoyang Sun Jiwen Jiang Yongcheng Zeng Xinjian Wu ... Haoyang Li Lei Chen Lionel M. Ni Hongzhi Zhang Jun Wang 168 0 0 15 Mar 2025
Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Models Kefan Song Jin Yao Runnan Jiang Rohan Chandra Shangtong Zhang ALM 46 0 0 10 Mar 2025
UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality Zelei Cheng Xin-Qiang Cai Yuting Tang Pushi Zhang Boming Yang Xinyu Xing Xinyu Xing 46 0 0 10 Mar 2025
DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models Ruizhe Chen Wenhao Chai Zhifei Yang Xiaotian Zhang Qiufeng Wang Tony Q. S. Quek Soujanya Poria Zuozhu Liu 50 0 0 06 Mar 2025
PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation Yuxuan Liu 45 0 0 03 Mar 2025
Accelerating Unbiased LLM Evaluation via Synthetic Feedback Zhaoyi Zhou Yuda Song Andrea Zanette ALM 73 0 0 14 Feb 2025
MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation Haibo Tong Zhaoyang Wang Zhengzhang Chen Haonian Ji Shi Qiu ... Peng Xia Mingyu Ding Rafael Rafailov Chelsea Finn Huaxiu Yao EGVM VGen 102 2 0 03 Feb 2025
Diverse Preference Optimization Jack Lanchantin Angelica Chen S. Dhuliawala Ping Yu Jason Weston Sainbayar Sukhbaatar Ilia Kulikov 93 4 0 30 Jan 2025
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling Bradley Brown Jordan Juravsky Ryan Ehrlich Ronald Clark Quoc V. Le Christopher Ré Azalia Mirhoseini ALM LRM 87 220 0 03 Jan 2025
Reinforcement Learning Enhanced LLMs: A Survey Shuhe Wang Shengyu Zhang Jingyang Zhang Runyi Hu Xiaoya Li Tianwei Zhang Jiwei Li Fei Wu G. Wang Eduard H. Hovy OffRL 134 7 0 05 Dec 2024
Self-Generated Critiques Boost Reward Modeling for Language Models Yue Yu Zhengxing Chen Aston Zhang L Tan Chenguang Zhu ... Suchin Gururangan Chao-Yue Zhang Melanie Kambadur Dhruv Mahajan Rui Hou LRM ALM 96 16 0 25 Nov 2024
Interpreting Language Reward Models via Contrastive Explanations Junqi Jiang Tom Bewley Saumitra Mishra Freddy Lecue Manuela Veloso 74 0 0 25 Nov 2024
Stronger Models are NOT Stronger Teachers for Instruction Tuning Zhangchen Xu Fengqing Jiang Luyao Niu Bill Yuchen Lin Radha Poovendran ALM 53 5 0 11 Nov 2024
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF Heyang Zhao Chenlu Ye Quanquan Gu Tong Zhang OffRL 57 3 0 07 Nov 2024
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following Gabrielle Kaili-May Liu Bowen Shi Avi Caciularu Idan Szpektor Arman Cohan 72 4 0 30 Oct 2024
Cross-lingual Transfer of Reward Models in Multilingual Alignment Jiwoo Hong Noah Lee Rodrigo Martínez-Castaño César Rodríguez James Thorne 48 4 0 23 Oct 2024
Taming Overconfidence in LLMs: Reward Calibration in RLHF Jixuan Leng Chengsong Huang Banghua Zhu Jiaxin Huang 34 7 0 13 Oct 2024
RMB: Comprehensively Benchmarking Reward Models in LLM Alignment Enyu Zhou Guodong Zheng Binghui Wang Zhiheng Xi Shihan Dou ... Yurong Mou Rui Zheng Tao Gui Qi Zhang Xuanjing Huang ALM 59 18 0 13 Oct 2024
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs Shenao Zhang Zhihan Liu Boyi Liu Yuhang Zhang Yingxiang Yang Y. Liu Liyu Chen Tao Sun Ziyi Wang 98 3 0 10 Oct 2024
ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time Yi Ding Bolian Li Ruqi Zhang MLLM 72 6 0 09 Oct 2024
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF Zhaolin Gao Wenhao Zhan Jonathan D. Chang Gokul Swamy Kianté Brantley Jason D. Lee Wen Sun OffRL 58 3 0 06 Oct 2024
RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization Hanyang Zhao Genta Indra Winata Anirban Das Shi-Xiong Zhang D. Yao Wenpin Tang Sambit Sahu 54 5 0 05 Oct 2024
TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation Jonathan Cook Tim Rocktaschel Jakob Foerster Dennis Aumiller Alex Wang ALM 34 10 0 04 Oct 2024
MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences Genta Indra Winata David Anugraha Lucky Susanto Garry Kuwanto Derry Wijaya 37 7 0 03 Oct 2024
HelpSteer2-Preference: Complementing Ratings with Preferences Zhilin Wang Alexander Bukharin Olivier Delalleau Daniel Egert Gerald Shen Jiaqi Zeng Oleksii Kuchaiev Yi Dong ALM 44 41 0 02 Oct 2024
Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown Xingzhou Lou Dong Yan Wei Shen Yuzi Yan Jian Xie Junge Zhang 47 22 0 01 Oct 2024
RRM: Robust Reward Model Training Mitigates Reward Hacking Tianqi Liu Wei Xiong Jie Jessie Ren Lichang Chen Junru Wu ... Yuan Liu Bilal Piot Abe Ittycheriah Aviral Kumar Mohammad Saleh AAML 56 13 0 20 Sep 2024
Aligning Language Models Using Follow-up Likelihood as Reward Signal Chen Zhang Dading Chong Feng Jiang Chengguang Tang Anningzhe Gao Guohua Tang Haizhou Li ALM 33 2 0 20 Sep 2024
From Lists to Emojis: How Format Bias Affects Model Alignment Xuanchang Zhang Wei Xiong Lichang Chen Dinesh Manocha Heng Huang Tong Zhang ALM 35 11 0 18 Sep 2024
Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models Wenxuan Zhang Philip H. S. Torr Mohamed Elhoseiny Adel Bibi 83 9 0 27 Aug 2024
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing Zhangchen Xu Fengqing Jiang Luyao Niu Yuntian Deng Radha Poovendran Yejin Choi Bill Yuchen Lin SyDa 34 120 0 12 Jun 2024
AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligence Minbeom Kim Hwanhee Lee Joonsuk Park Hwaran Lee Kyomin Jung 32 1 0 18 Apr 2024
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Hassan Awadallah Tengyang Xie 152 114 0 04 Apr 2024
RewardBench: Evaluating Reward Models for Language Modeling Nathan Lambert Valentina Pyatkin Jacob Morrison Lester James Validad Miranda Bill Yuchen Lin ... Sachin Kumar Tom Zick Yejin Choi Noah A. Smith Hanna Hajishirzi ALM 82 214 0 20 Mar 2024
CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences Martin Weyssow Aton Kamanda H. Sahraoui ALM 64 32 0 14 Mar 2024
ODIN: Disentangled Reward Mitigates Hacking in RLHF Lichang Chen Chen Zhu Davit Soselia Jiuhai Chen Dinesh Manocha Tom Goldstein Heng-Chiao Huang M. Shoeybi Bryan Catanzaro AAML 47 51 0 11 Feb 2024
Defining and Characterizing Reward Hacking Joar Skalse Nikolaus H. R. Howe Dmitrii Krasheninnikov David M. Krueger 59 55 0 27 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,953 0 04 Mar 2022
$Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information$ Understanding Dataset Difficulty with $\mathcal{V}$ -Usable Information Kawin Ethayarajh Yejin Choi Swabha Swayamdipta 167 157 0 16 Oct 2021