v1v2 (latest)

ORPO: Monolithic Preference Optimization without Reference Model

12 March 2024

Papers citing "ORPO: Monolithic Preference Optimization without Reference Model"

50 / 52 papers shown

Title
Rethinking Direct Preference Optimization in Diffusion Models Junyong Kang Seohyun Lim Kyungjune Baek Hyunjung Shim 764 0 0 24 May 2025
Bridging Supervised Learning and Reinforcement Learning in Math Reasoning Huayu Chen Kaiwen Zheng Qinsheng Zhang Ganqu Cui Yin Cui Haotian Ye Tsung-Yi Lin Ming-Yu Liu Jun Zhu Haoxiang Wang OffRL LRM 236 3 0 23 May 2025
SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment Wenqiao Zhu Ji Liu Lulu Wang Jun Wu Yulun Zhang 82 0 0 18 May 2025
Bielik 11B v2 Technical Report Krzysztof Ociepa Łukasz Flis Krzysztof Wróbel Adrian Gwoździej Remigiusz Kinas 77 0 0 05 May 2025
From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs Jiliang Ni Jiachen Pu Zhongyi Yang Kun Zhou Hui Wang Xiaoliang Xiao Dakui Wang Xin Li Jingfeng Luo Conggang Hu 98 0 0 18 Apr 2025
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion Longguang Zhong Fanqi Wan Ziyi Yang Guosheng Liang Tianyuan Shi Xiaojun Quan MoMe 104 0 0 09 Apr 2025
SOLAR: Scalable Optimization of Large-scale Architecture for Reasoning Chen Li Yinyi Luo Anudeep Bolimera Uzair Ahmed Siyang Song Hrishikesh Gokhale Marios Savvides LRM AI4CE 82 1 0 06 Mar 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Ziyang Chen Mingxiao Li Shangsong Liang Zhaochun Ren V. Honavar 219 11 0 21 Feb 2025
Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees Yongtao Wu Luca Viano Yihang Chen Zhenyu Zhu Kimon Antonakopoulos Quanquan Gu Volkan Cevher 135 1 0 18 Feb 2025
LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data Cehao Yang Xueyuan Lin Chengjin Xu Xuhui Jiang Shengjie Ma Aofan Liu Hui Xiong Jian Guo LRM 51 2 0 18 Feb 2025
HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humor Zihui Wu Haichang Gao Jiacheng Luo Zhaoxiang Liu 126 0 0 23 Jan 2025
Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteria Joonwon Jang Jaehee Kim Wonbin Kweon Seonghyeon Lee Hwanjo Yu LRM 121 0 0 30 Dec 2024
REFA: Reference Free Alignment for multi-preference optimization Taneesh Gupta Rahul Madhavan Xuchao Zhang Chetan Bansal Saravan Rajmohan 135 1 0 20 Dec 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 127 91 1 15 Nov 2024
Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings Miguel Moura Ramos Tomás Almeida Daniel Vareta Filipe Azevedo Sweta Agrawal Patrick Fernandes André F. T. Martins 104 4 0 08 Nov 2024
TODO: Enhancing LLM Alignment with Ternary Preferences Yuxiang Guo Lu Yin Bo Jiang Jiaqi Zhang 109 3 0 02 Nov 2024
$f$ -PO: Generalizing Preference Optimization with $f$ -divergence Minimization Jiaqi Han Mingjian Jiang Yuxuan Song J. Leskovec Stefano Ermon 103 6 0 29 Oct 2024
UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function Zhichao Wang Bin Bi Z. Zhu Xiangbo Mao Jun Wang Shiyu Wang CLL 71 1 0 28 Oct 2024
Cross-lingual Transfer of Reward Models in Multilingual Alignment Jiwoo Hong Noah Lee Rodrigo Martínez-Castaño César Rodríguez James Thorne 128 5 0 23 Oct 2024
Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning H. Fernando Han Shen Parikshit Ram Yi Zhou Horst Samulowitz Nathalie Baracaldo Tianyi Chen CLL 128 4 0 20 Oct 2024
Taming Overconfidence in LLMs: Reward Calibration in RLHF Jixuan Leng Chengsong Huang Banghua Zhu Jiaxin Huang 100 16 0 13 Oct 2024
Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both Abhijnan Nath Changsoo Jung Ethan Seefried Nikhil Krishnaswamy 457 4 0 11 Oct 2024
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees Weibin Liao Xu Chu Yasha Wang LRM 110 8 0 10 Oct 2024
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF Zhaolin Gao Wenhao Zhan Jonathan D. Chang Gokul Swamy Kianté Brantley Jason D. Lee Wen Sun OffRL 132 7 0 06 Oct 2024
Aligning Language Models Using Follow-up Likelihood as Reward Signal Chen Zhang Dading Chong Feng Jiang Chengguang Tang Anningzhe Gao Guohua Tang Haizhou Li ALM 81 2 0 20 Sep 2024
From Lists to Emojis: How Format Bias Affects Model Alignment Xuanchang Zhang Wei Xiong Lichang Chen Dinesh Manocha Heng Huang Tong Zhang ALM 94 14 0 18 Sep 2024
Propaganda is all you need Paul Kronlund-Drouault 105 0 0 13 Sep 2024
RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data Chenglong Wang Yang Gan Yifu Huo Yongyu Mu Murun Yang ... Chunliang Zhang Tongran Liu Quan Du Di Yang Jingbo Zhu VLM 118 6 0 22 Aug 2024
Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization Yuxin Jiang Bo Huang Yufei Wang Xingshan Zeng Liangyou Li Yasheng Wang Xin Jiang Lifeng Shang Ruiming Tang Wei Wang 99 7 0 14 Aug 2024
PORT: Preference Optimization on Reasoning Traces Salem Lahlou Abdalgader Abubaker Hakim Hacid LRM 94 5 0 23 Jun 2024
Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency Leonidas Gee Milan Gritta Gerasimos Lampouras Ignacio Iacobacci 78 10 0 18 Jun 2024
PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration Huiping Zhuang Jianwei Wang Zhengdong Lu Huiping Zhuang Haoran Li Huiping Zhuang Cen Chen RALM KELM 93 8 0 03 Jun 2024
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts Minghao Wu Jiahao Xu Yulin Yuan Gholamreza Haffari Longyue Wang Weihua Luo Kaifu Zhang LLMAG 168 27 0 20 May 2024
Disentangling Length from Quality in Direct Preference Optimization Ryan Park Rafael Rafailov Stefano Ermon Chelsea Finn ALM 91 145 0 28 Mar 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 270 565 0 02 Feb 2024
Aligning Neural Machine Translation Models: Human Feedback in Training and Inference Miguel Moura Ramos Patrick Fernandes António Farinhas André F. T. Martins ALM 89 18 0 15 Nov 2023
Fine-tuning Language Models for Factuality Katherine Tian Eric Mitchell Huaxiu Yao Christopher D. Manning Chelsea Finn KELM HILM SyDa 75 184 0 14 Nov 2023
Zephyr: Direct Distillation of LM Alignment Lewis Tunstall E. Beeching Nathan Lambert Nazneen Rajani Kashif Rasul ... Nathan Habib Nathan Sarrazin Omar Sanseviero Alexander M. Rush Thomas Wolf ALM 102 397 0 25 Oct 2023
A General Theoretical Paradigm to Understand Learning from Human Preferences M. G. Azar Mark Rowland Bilal Piot Daniel Guo Daniele Calandriello Michal Valko Rémi Munos 176 645 0 18 Oct 2023
Understanding the Effects of RLHF on LLM Generalisation and Diversity Robert Kirk Ishita Mediratta Christoforos Nalmpantis Jelena Luketina Eric Hambro Edward Grefenstette Roberta Raileanu AI4CE ALM 170 149 0 10 Oct 2023
Reinforced Self-Training (ReST) for Language Modeling Çağlar Gülçehre T. Paine S. Srinivasan Ksenia Konyushkova L. Weerts ... Chenjie Gu Wolfgang Macherey Arnaud Doucet Orhan Firat Nando de Freitas OffRL 121 308 0 17 Aug 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 408 4,422 0 09 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 387 4,139 0 29 May 2023
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment Hanze Dong Wei Xiong Deepanshu Goyal Yihan Zhang Winnie Chow Boyao Wang Shizhe Diao Jipeng Zhang Kashun Shum Tong Zhang ALM 94 467 0 13 Apr 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 883 13,176 0 04 Mar 2022
Finetuned Language Models Are Zero-Shot Learners Jason W. Wei Maarten Bosma Vincent Zhao Kelvin Guu Adams Wei Yu Brian Lester Nan Du Andrew M. Dai Quoc V. Le ALM UQCV 217 3,782 0 03 Sep 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown Basel Alomair Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 509 1,946 0 14 Dec 2020
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models Samuel Gehman Suchin Gururangan Maarten Sap Yejin Choi Noah A. Smith 163 1,214 0 24 Sep 2020
Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering Vikas Yadav Steven Bethard Mihai Surdeanu 114 77 0 17 Nov 2019
Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training Margaret Li Stephen Roller Ilia Kulikov Sean Welleck Y-Lan Boureau Kyunghyun Cho Jason Weston 75 183 0 10 Nov 2019