Self-Exploring Language Models: Active Preference Elicitation for Online Alignment

29 May 2024

Papers citing "Self-Exploring Language Models: Active Preference Elicitation for Online Alignment"

27 / 27 papers shown

Title
Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF Syrine Belakaria Joshua Kazdan Charles Marx Chris Cundy W. Neiswanger Sanmi Koyejo Barbara Engelhardt Stefano Ermon 34 0 0 28 Mar 2025
Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective Jiawei Huang Bingcong Li Christoph Dann Niao He OffRL 82 0 0 26 Feb 2025
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF Shicong Cen Jincheng Mei Katayoon Goshvadi Hanjun Dai Tong Yang Sherry Yang Dale Schuurmans Yuejie Chi Bo Dai OffRL 62 23 0 20 Feb 2025
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang Qirun Dai Hao Peng ALM 115 3 0 06 Feb 2025
Understanding the Logic of Direct Preference Alignment through Logic Kyle Richardson Vivek Srikumar Ashish Sabharwal 85 2 0 23 Dec 2024
Online Learning from Strategic Human Feedback in LLM Fine-Tuning Shugang Hao Lingjie Duan 84 3 0 22 Dec 2024
Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration Avinandan Bose Zhihan Xiong Aadirupa Saha S. Du Maryam Fazel 71 1 0 13 Dec 2024
DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs Zhihan Liu Shenao Zhang Yongfei Liu Boyi Liu Yingxiang Yang Zhaoran Wang 111 2 0 20 Nov 2024
Sample-Efficient Alignment for LLMs Zichen Liu Changyu Chen Chao Du Wee Sun Lee Min-Bin Lin 36 3 0 03 Nov 2024
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization Noam Razin Sadhika Malladi Adithya Bhaskar Danqi Chen Sanjeev Arora Boris Hanin 91 14 0 11 Oct 2024
Reward-Augmented Data Enhances Direct Preference Alignment of LLMs Shenao Zhang Zhihan Liu Boyi Liu Y. Zhang Yingxiang Yang Y. Liu Liyu Chen Tao Sun Z. Wang 93 3 0 10 Oct 2024
Can Language Models Reason about Individualistic Human Values and Preferences? Liwei Jiang Taylor Sorensen Sydney Levine Yejin Choi 33 7 0 04 Oct 2024
Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization Ruijie Xu Zhihan Liu Yongfei Liu Shipeng Yan Zhaoran Wang Zhi-Li Zhang Xuming He ALM 35 1 0 26 Sep 2024
From Lists to Emojis: How Format Bias Affects Model Alignment Xuanchang Zhang Wei Xiong Lichang Chen Tianyi Zhou Heng Huang Tong Zhang ALM 33 11 0 18 Sep 2024
Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning S. Poddar Yanming Wan Hamish Ivison Abhishek Gupta Natasha Jaques 32 35 0 19 Aug 2024
Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift Seongho Son William Bankes Sayak Ray Chowdhury Brooks Paige Ilija Bogunovic 34 4 0 26 Jul 2024
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Haoxiang Wang Wei Xiong Tengyang Xie Han Zhao Tong Zhang 46 136 0 18 Jun 2024
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF Tengyang Xie Dylan J. Foster Akshay Krishnamurthy Corby Rosset Ahmed Hassan Awadallah Alexander Rakhlin 43 33 0 31 May 2024
RLHF Workflow: From Reward Modeling to Online RLHF Hanze Dong Wei Xiong Bo Pang Haoxiang Wang Han Zhao Yingbo Zhou Nan Jiang Doyen Sahoo Caiming Xiong Tong Zhang OffRL 27 95 0 13 May 2024
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Hassan Awadallah Tengyang Xie 149 114 0 04 Apr 2024
Multi-modal preference alignment remedies regression of visual instruction tuning on language model Shengzhi Li Rongyu Lin Shichao Pei 40 20 0 16 Feb 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 170 444 0 02 Feb 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 237 298 0 18 Jan 2024
Language Model Alignment with Elastic Reset Michael Noukhovitch Samuel Lavoie Florian Strub Aaron Courville KELM 89 25 0 06 Dec 2023
Instruction Tuning with GPT-4 Baolin Peng Chunyuan Li Pengcheng He Michel Galley Jianfeng Gao SyDa ALM LM&MA 159 579 0 06 Apr 2023
Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction Martin Josifoski Marija Sakota Maxime Peyrard Robert West SyDa 56 78 0 07 Mar 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,915 0 04 Mar 2022