PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation

3 March 2025

Papers citing "PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation"

14 / 14 papers shown

Title
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework Jian Hu Xibin Wu Weixun Wang OpenLLMAI Team Dehao Zhang Yu Cao AI4CE VLM 106 130 0 20 May 2024
ORPO: Monolithic Preference Optimization without Reference Model Jiwoo Hong Noah Lee James Thorne OSLM 98 266 0 12 Mar 2024
Provably Robust DPO: Aligning Language Models with Noisy Feedback Sayak Ray Chowdhury Anush Kini Nagarajan Natarajan 78 70 0 01 Mar 2024
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards Haoxiang Wang Yong Lin Wei Xiong Rui Yang Shizhe Diao Shuang Qiu Han Zhao Tong Zhang 107 87 0 28 Feb 2024
HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition Yuxuan Liu Tianchi Yang Shaohan Huang Zihan Zhang Haizhen Huang Furu Wei Weiwei Deng Feng Sun Qi Zhang 51 16 0 24 Feb 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 275 569 0 02 Feb 2024
Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization Zhanhui Zhou Jie Liu Chao Yang Jing Shao Yu Liu Xiangyu Yue Wanli Ouyang Yu Qiao 73 61 0 05 Oct 2023
Reward Model Ensembles Help Mitigate Overoptimization Thomas Coste Usman Anwar Robert Kirk David M. Krueger NoLa ALM 88 138 0 04 Oct 2023
Reinforced Self-Training (ReST) for Language Modeling Çağlar Gülçehre T. Paine S. Srinivasan Ksenia Konyushkova L. Weerts ... Chenjie Gu Wolfgang Macherey Arnaud Doucet Orhan Firat Nando de Freitas OffRL 125 309 0 17 Aug 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 389 4,163 0 29 May 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,748 0 15 Mar 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 888 13,207 0 04 Mar 2022
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 547 19,296 0 20 Jul 2017
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.1K 150,364 0 22 Dec 2014