v1v2 (latest)

A General Theoretical Paradigm to Understand Learning from Human Preferences

18 October 2023

Bilal Piot

Daniele Calandriello

Papers citing "A General Theoretical Paradigm to Understand Learning from Human Preferences"

50 / 98 papers shown

Title
Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models Y. Zhang Yu Yu Bo Tang Yu Zhu Chuxiong Sun ... Jie Hu Zipeng Xie Zhiyu Li Feiyu Xiong Edward Chung 91 0 0 26 May 2025
Rethinking Direct Preference Optimization in Diffusion Models Junyong Kang Seohyun Lim Kyungjune Baek Hyunjung Shim 770 0 0 24 May 2025
Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning Yutong Chen Jiandong Gao Ji Wu ALM 201 0 0 23 May 2025
Bridging Supervised Learning and Reinforcement Learning in Math Reasoning Huayu Chen Kaiwen Zheng Qinsheng Zhang Ganqu Cui Yin Cui Haotian Ye Tsung-Yi Lin Ming-Yu Liu Jun Zhu Haoxiang Wang OffRL LRM 241 3 0 23 May 2025
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions Chaochen Gao Xing Wu Zijia Lin Debing Zhang Songlin Hu SyDa 207 0 0 22 May 2025
On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding Haoyuan Wu Rui Ming Jilong Gao Hangyu Zhao Xueyi Chen Yikai Yang Haisheng Zheng Zhuolun He Bei Yu 117 0 0 19 May 2025
SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment Wenqiao Zhu Ji Liu Lulu Wang Jun Wu Yulun Zhang 90 0 0 18 May 2025
InfoPO: On Mutual Information Maximization for Large Language Model Alignment Teng Xiao Zhen Ge Sujay Sanghavi Tian Wang Julian Katz-Samuels Marc Versage Qingjun Cui Trishul Chilimbi 193 1 0 13 May 2025
Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models Rei Higuchi Taiji Suzuki 109 1 0 12 May 2025
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL Jiarui Yao Yifan Hao Hanning Zhang Hanze Dong Wei Xiong Nan Jiang Tong Zhang LRM 158 2 0 05 May 2025
2D-Curri-DPO: Two-Dimensional Curriculum Learning for Direct Preference Optimization Mengyang Li Zhong Zhang 71 1 0 10 Apr 2025
Information-Theoretic Reward Decomposition for Generalizable RLHF Liyuan Mao Haoran Xu Amy Zhang Weinan Zhang Chenjia Bai 109 0 0 08 Apr 2025
Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use Anna Goldie Azalia Mirhoseini Hao Zhou Irene Cai Christopher D. Manning SyDa OffRL ReLM LRM 183 11 0 07 Apr 2025
Reasoning Beyond Limits: Advances and Open Problems for LLMs M. Ferrag Norbert Tihanyi Merouane Debbah ELM OffRL LRM AI4CE 414 4 0 26 Mar 2025
Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs Nicolas Le Roux Marc G. Bellemare Jonathan Lebensold Arnaud Bergeron Joshua Greaves Alex Fréchette Carolyne Pelletier Eric Thibodeau-Laufer Sándor Toth Sam Work OffRL 162 6 0 18 Mar 2025
Preference-Based Alignment of Discrete Diffusion Models Umberto Borso Davide Paglieri Jude Wells Tim Rocktaschel 108 3 0 11 Mar 2025
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective Jiawei Huang Bingcong Li Christoph Dann Niao He OffRL 262 3 0 26 Feb 2025
Improving LLM General Preference Alignment via Optimistic Online Mirror Descent Yuheng Zhang Dian Yu Tao Ge Linfeng Song Zhichen Zeng Haitao Mi Nan Jiang Dong Yu 124 4 0 24 Feb 2025
Post-edits Are Preferences Too Nathaniel Berger Stefan Riezler M. Exel Matthias Huck 126 2 0 24 Feb 2025
Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding Tian Jin Ellie Y. Cheng Zack Ankner Nikunj Saunshi Blake M. Elias Amir Yazdanbakhsh Jonathan Ragan-Kelley Suvinay Subramanian Michael Carbin 130 5 0 24 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Ziyang Chen Mingxiao Li Shangsong Liang Zhaochun Ren V. Honavar 254 11 0 21 Feb 2025
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF Shicong Cen Jincheng Mei Katayoon Goshvadi Hanjun Dai Tong Yang Sherry Yang Dale Schuurmans Yuejie Chi Bo Dai OffRL 146 37 0 20 Feb 2025
Simplify RLHF as Reward-Weighted SFT: A Variational Method Yuhao Du Zehan Li Pengyu Cheng Zhihong Chen Yuejiao Xie Xiang Wan Anningzhe Gao 110 1 0 20 Feb 2025
Faster WIND: Accelerating Iterative Best-of- $N$ Distillation for LLM Alignment Tong Yang Jincheng Mei H. Dai Zixin Wen Shicong Cen Dale Schuurmans Yuejie Chi Bo Dai 120 4 0 20 Feb 2025
Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees Yongtao Wu Luca Viano Yihang Chen Zhenyu Zhu Kimon Antonakopoulos Quanquan Gu Volkan Cevher 166 1 0 18 Feb 2025
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization Shuo Xing Yuping Wang Peiran Li Ruizheng Bai Yansen Wang Chan-wei Hu Chengxuan Qian Huaxiu Yao Zhengzhong Tu 181 8 0 18 Feb 2025
Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models Yingshui Tan Yilei Jiang Yongbin Li Qingbin Liu Xingyuan Bu Wenbo Su Xiangyu Yue Xiaoyong Zhu Bo Zheng ALM 143 6 0 17 Feb 2025
Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training Fenghua Weng Jian Lou Jun Feng Minlie Huang Wenjie Wang AAML 157 2 0 17 Feb 2025
Design Considerations in Offline Preference-based RL Alekh Agarwal Christoph Dann T. V. Marinov OffRL 105 1 0 08 Feb 2025
Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance Shehzeen Samarah Hussain Paarth Neekhara Xuesong Yang Edresson Casanova Subhankar Ghosh Mikyas T. Desta Roy Fejgin Rafael Valle Jason Chun Lok Li 136 5 0 07 Feb 2025
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang Qirun Dai Hao Peng ALM 209 7 0 06 Feb 2025
LiPO: Listwise Preference Optimization through Learning-to-Rank Tianqi Liu Zhen Qin Junru Wu Jiaming Shen Misha Khalman ... Mohammad Saleh Simon Baumgartner Jialu Liu Peter J. Liu Xuanhui Wang 321 60 0 28 Jan 2025
Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning Hao Sun M. Schaar 152 18 0 28 Jan 2025
Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion Yannis Flet-Berliac Nathan Grinsztajn Florian Strub Bill Wu Eugene Choi ... Arash Ahmadian Yash Chandak M. G. Azar Olivier Pietquin Matthieu Geist OffRL 155 10 0 17 Jan 2025
An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems Hashmath Shaik Alex Doboli OffRL ELM 445 0 0 31 Dec 2024
Geometric-Averaged Preference Optimization for Soft Preference Labels Hiroki Furuta Kuang-Huei Lee Shixiang Shane Gu Y. Matsuo Aleksandra Faust Heiga Zen Izzeddin Gur 137 13 0 31 Dec 2024
Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteria Joonwon Jang Jaehee Kim Wonbin Kweon Seonghyeon Lee Hwanjo Yu LRM 145 2 0 30 Dec 2024
Understanding the Logic of Direct Preference Alignment through Logic Kyle Richardson Vivek Srikumar Ashish Sabharwal 209 2 0 23 Dec 2024
REFA: Reference Free Alignment for multi-preference optimization Taneesh Gupta Rahul Madhavan Xuchao Zhang Chetan Bansal Saravan Rajmohan 163 1 0 20 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 213 2 0 01 Dec 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 129 92 1 15 Nov 2024
Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching A. Jain Harley Wiltzer Jesse Farebrother Irina Rish Glen Berseth Sanjiban Choudhury 122 2 0 11 Nov 2024
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF Heyang Zhao Chenlu Ye Quanquan Gu Tong Zhang OffRL 234 6 0 07 Nov 2024
TODO: Enhancing LLM Alignment with Ternary Preferences Yuxiang Guo Lu Yin Bo Jiang Jiaqi Zhang 116 3 0 02 Nov 2024
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence Minimization Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 111 6 0 29 Oct 2024
UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function Zhichao Wang Bin Bi Z. Zhu Xiangbo Mao Jun Wang Shiyu Wang CLL 77 1 0 28 Oct 2024
Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning H. Fernando Han Shen Parikshit Ram Yi Zhou Horst Samulowitz Nathalie Baracaldo Tianyi Chen CLL 164 4 0 20 Oct 2024
Preference Optimization with Multi-Sample Comparisons Chaoqi Wang Zhuokai Zhao Chen Zhu Karthik Abinav Sankararaman Michal Valko ... Zhaorun Chen Madian Khabsa Yuxin Chen Hao Ma Sinong Wang 139 9 0 16 Oct 2024
Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leaps Han Wang Yilin Zhao Dian Li Xiaohan Wang Gang Liu Xuguang Lan Haoran Wang LRM 152 1 0 14 Oct 2024
Taming Overconfidence in LLMs: Reward Calibration in RLHF Jixuan Leng Chengsong Huang Banghua Zhu Jiaxin Huang 112 16 0 13 Oct 2024