v1v2v3 (latest)

Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization

14 August 2024

Yasheng Wang

Lifeng Shang

Ruiming Tang

Wei Wang

ArXiv (abs)PDF HTML

Papers citing "Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization"

44 / 44 papers shown

Title
From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignment Jing Ye Lu Xiang Yaping Zhang Chengqing Zong 61 0 0 22 May 2025
Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction Yuxin Jiang Yijiao Wang Chuhan Wu Xinyi Dai Yan Xu ... Yucheng Wang Xin Jiang Lifeng Shang Ruiming Tang Wenjie Wang 110 0 0 22 Apr 2025
Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Time Jiazheng Li Yuxiang Zhou Junru Lu Gladys Tyen Lin Gui Cesare Aloisi Yulan He LRM 91 3 0 26 Feb 2025
2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision Shilong Li Yancheng He Hui Huang Xingyuan Bu Qingbin Liu Hangyu Guo Weixun Wang Jihao Gu Wenbo Su Bo Zheng 84 7 0 25 Oct 2024
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References Qiyuan Zhang Yufei Wang Tiezheng YU Yuxin Jiang Chuhan Wu ... Xin Jiang Lifeng Shang Ruiming Tang Fuyuan Lyu Chen Ma 73 7 0 07 Oct 2024
A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More Zhichao Wang Bin Bi Shiva K. Pentyala Kiran Ramnath Sougata Chaudhuri ... Z. Zhu Xiang-Bo Mao S. Asur Na Na Cheng OffRL 79 58 0 23 Jul 2024
Low-Redundant Optimization for Large Language Model Alignment Zhipeng Chen Kun Zhou Wayne Xin Zhao Jingyuan Wang Ji-Rong Wen 76 3 0 18 Jun 2024
SimPO: Simple Preference Optimization with a Reference-Free Reward Yu Meng Mengzhou Xia Danqi Chen 142 488 0 23 May 2024
Token-level Direct Preference Optimization Yongcheng Zeng Guoqing Liu Weiyu Ma Ning Yang Haifeng Zhang Jun Wang 97 64 0 18 Apr 2024
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators Yann Dubois Balázs Galambosi Percy Liang Tatsunori Hashimoto ALM 133 402 0 06 Apr 2024
Disentangling Length from Quality in Direct Preference Optimization Ryan Park Rafael Rafailov Stefano Ermon Chelsea Finn ALM 93 145 0 28 Mar 2024
ORPO: Monolithic Preference Optimization without Reference Model Jiwoo Hong Noah Lee James Thorne OSLM 96 266 0 12 Mar 2024
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification Ekaterina Fadeeva Aleksandr Rubashevskii Artem Shelmanov Sergey Petrakov Haonan Li ... Gleb Kuzmin Alexander Panchenko Timothy Baldwin Preslav Nakov Maxim Panov HILM 89 56 0 07 Mar 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 270 565 0 02 Feb 2024
Dense Reward for Free in Reinforcement Learning from Human Feedback Alex J. Chan Hao Sun Samuel Holt M. Schaar 73 42 0 01 Feb 2024
Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint Zhipeng Chen Kun Zhou Wayne Xin Zhao Junchen Wan Fuzheng Zhang Di Zhang Ji-Rong Wen KELM 90 35 0 11 Jan 2024
CoachLM: Automatic Instruction Revisions Improve the Data Quality in LLM Instruction Tuning Yilun Liu Shimin Tao Xiaofeng Zhao Ming Zhu Wenbing Ma ... Min Zhang Hongxia Ma Li Zhang Hao Yang Yanfei Jiang 78 13 0 22 Nov 2023
Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment Geyang Guo Ranchi Zhao Tianyi Tang Wayne Xin Zhao Ji-Rong Wen ALM 85 32 0 07 Nov 2023
Zephyr: Direct Distillation of LM Alignment Lewis Tunstall E. Beeching Nathan Lambert Nazneen Rajani Kashif Rasul ... Nathan Habib Nathan Sarrazin Omar Sanseviero Alexander M. Rush Thomas Wolf ALM 107 398 0 25 Oct 2023
A General Theoretical Paradigm to Understand Learning from Human Preferences M. G. Azar Mark Rowland Bilal Piot Daniel Guo Daniele Calandriello Michal Valko Rémi Munos 180 645 0 18 Oct 2023
Mistral 7B Albert Q. Jiang Alexandre Sablayrolles A. Mensch Chris Bamford Devendra Singh Chaplot ... Teven Le Scao Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LRM 104 2,238 0 10 Oct 2023
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models L. Yu Weisen Jiang Han Shi Jincheng Yu Zhengying Liu Yu Zhang James T. Kwok Zheng Li Adrian Weller Weiyang Liu OSLM LRM 104 395 0 21 Sep 2023
Statistical Rejection Sampling Improves Preference Optimization Tianqi Liu Yao-Min Zhao Rishabh Joshi Misha Khalman Mohammad Saleh Peter J. Liu Jialu Liu 131 249 0 13 Sep 2023
Efficient RLHF: Reducing the Memory Usage of PPO Michael Santacroce Yadong Lu Han Yu Yuan-Fang Li Yelong Shen 64 32 0 01 Sep 2023
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback Stephen Casper Xander Davies Claudia Shi T. Gilbert Jérémy Scheurer ... Erdem Biyik Anca Dragan David M. Krueger Dorsa Sadigh Dylan Hadfield-Menell ALM OffRL 135 530 0 27 Jul 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 399 12,076 0 18 Jul 2023
Preference-grounded Token-level Guidance for Language Model Fine-tuning Shentao Yang Shujian Zhang Congying Xia Yihao Feng Caiming Xiong Mi Zhou 112 28 0 01 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 389 4,163 0 29 May 2023
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations Ning Ding Yulin Chen Bokai Xu Yujia Qin Zhi Zheng Shengding Hu Zhiyuan Liu Maosong Sun Bowen Zhou ALM 150 552 0 23 May 2023
Aligning Large Language Models through Synthetic Feedback Sungdong Kim Sanghwan Bae Jamin Shin Soyoung Kang Donghyun Kwak Kang Min Yoo Minjoon Seo ALM SyDa 121 70 0 23 May 2023
GPTScore: Evaluate as You Desire Jinlan Fu See-Kiong Ng Zhengbao Jiang Pengfei Liu LM&MA ALM ELM 163 289 0 08 Feb 2023
Self-Instruct: Aligning Language Models with Self-Generated Instructions Yizhong Wang Yeganeh Kordi Swaroop Mishra Alisa Liu Noah A. Smith Daniel Khashabi Hannaneh Hajishirzi ALM SyDa LRM 151 2,253 0 20 Dec 2022
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning Pan Lu Liang Qiu Kai-Wei Chang Ying Nian Wu Song-Chun Zhu Tanmay Rajpurohit Peter Clark Ashwin Kalyan ReLM LRM 171 296 0 29 Sep 2022
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen ... Jack Clark Sam McCandlish C. Olah Benjamin Mann Jared Kaplan 256 2,623 0 12 Apr 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 886 13,207 0 04 Mar 2022
A General Language Assistant as a Laboratory for Alignment Amanda Askell Yuntao Bai Anna Chen Dawn Drain Deep Ganguli ... Tom B. Brown Jack Clark Sam McCandlish C. Olah Jared Kaplan ALM 120 790 0 01 Dec 2021
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 350 4,596 0 27 Oct 2021
BARTScore: Evaluating Generated Text as Text Generation Weizhe Yuan Graham Neubig Pengfei Liu 126 849 0 22 Jun 2021
On Hallucination and Predictive Uncertainty in Conditional Language Generation Yijun Xiao Wenjie Wang HILM 159 192 0 28 Mar 2021
Measuring Mathematical Problem Solving With the MATH Dataset Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang Basel Alomair Jacob Steinhardt ReLM FaML 183 2,405 0 05 Mar 2021
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies Mor Geva Daniel Khashabi Elad Segal Tushar Khot Dan Roth Jonathan Berant RALM 347 737 0 06 Jan 2021
QASC: A Dataset for Question Answering via Sentence Composition Tushar Khot Peter Clark Michal Guerquin Peter Alexander Jansen Ashish Sabharwal CoGe 81 330 0 25 Oct 2019
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering Todor Mihaylov Peter Clark Tushar Khot Ashish Sabharwal 119 1,570 0 08 Sep 2018
Deep reinforcement learning from human preferences Paul Christiano Jan Leike Tom B. Brown Miljan Martic Shane Legg Dario Amodei 218 3,377 0 12 Jun 2017