SimPO: Simple Preference Optimization with a Reference-Free Reward

23 May 2024

Papers citing "SimPO: Simple Preference Optimization with a Reference-Free Reward"

50 / 121 papers shown

Title
HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages Ziyi Wang Jiaqi Zeng Olivier Delalleau Hoo-Chang Shin Felipe Soares Alexander Bukharin Ellie Evans Yi Dong Oleksii Kuchaiev 17 0 0 16 May 2025
Preference Optimization for Combinatorial Optimization Problems Mingjun Pan Guanquan Lin You-Wei Luo Bin Zhu Zhien Dai Lijun Sun Chun Yuan 23 0 0 13 May 2025
InfoPO: On Mutual Information Maximization for Large Language Model Alignment Teng Xiao Zhen Ge Sujay Sanghavi Tian Wang Julian Katz-Samuels Marc Versage Qingjun Cui Trishul M. Chilimbi 31 0 0 13 May 2025
On the Robustness of Reward Models for Language Model Alignment Jiwoo Hong Noah Lee Eunki Kim Guijin Son Woojin Chung Aman Gupta Shao Tang James Thorne 29 0 0 12 May 2025
ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoning Ziqing Qiao Yongheng Deng Jiali Zeng Dong Wang Lai Wei Fandong Meng Jie Zhou Ju Ren Yaoxue Zhang LRM 49 0 0 08 May 2025
LLAMAPIE: Proactive In-Ear Conversation Assistants Tuochao Chen Nicholas Batchelder Alisa Liu Noah A. Smith Shyamnath Gollakota 128 0 0 07 May 2025
Policy-labeled Preference Learning: Is Preference Enough for RLHF? Taehyun Cho Seokhun Ju Seungyub Han Dohyeong Kim Kyungjae Lee Jungwoo Lee OffRL 29 0 0 06 May 2025
Bielik 11B v2 Technical Report Krzysztof Ociepa Łukasz Flis Krzysztof Wróbel Adrian Gwoździej Remigiusz Kinas 34 0 0 05 May 2025
FairPO: Robust Preference Optimization for Fair Multi-Label Learning Soumen Kumar Mondal Akshit Varmora Prateek Chanda Ganesh Ramakrishnan 32 0 0 05 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 72 1 0 05 May 2025
A Survey on Progress in LLM Alignment from the Perspective of Reward Design Miaomiao Ji Yanqiu Wu Zhibin Wu Shoujin Wang Jian Yang Mark Dras Usman Naseem 39 0 0 05 May 2025
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning Tianjian Li Daniel Khashabi 55 0 0 05 May 2025
Improving Model Alignment Through Collective Intelligence of Open-Source LLMS Junlin Wang Roy Xie Shang Zhu Jue Wang Ben Athiwaratkun Bhuwan Dhingra S. Song Ce Zhang James Y. Zou ALM 31 0 0 05 May 2025
Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs Jinyan Su Jennifer Healey Preslav Nakov Claire Cardie LRM 150 0 0 30 Apr 2025
Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning Joykirat Singh Raghav Magazine Yash Pandya A. Nambi LLMAG KELM OffRL LRM 141 1 0 28 Apr 2025
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors Ren-Wei Liang Chin-Ting Hsu Chan-Hung Yu Saransh Agrawal Shih-Cheng Huang Shang-Tse Chen Kuan-Hao Huang Shao-Hua Sun 81 0 0 27 Apr 2025
Calibrating Translation Decoding with Quality Estimation on LLMs Di Wu Yibin Lei Christof Monz 70 0 0 26 Apr 2025
Parameter-Efficient Checkpoint Merging via Metrics-Weighted Averaging Shi Jie Yu Sehyun Choi MoMe 56 0 0 23 Apr 2025
AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization Jinda Lu Jinghan Li Yuan Gao Junkang Wu Jiancan Wu X. Wang Xiangnan He 118 0 0 22 Apr 2025
Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model Junshu Pan Wei Shen Shulin Huang Qiji Zhou Yue Zhang 69 0 0 22 Apr 2025
From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs Jiliang Ni Jiachen Pu Zhongyi Yang Kun Zhou Hui Wang Xiaoliang Xiao Dakui Wang Xin Li Jingfeng Luo Conggang Hu 34 0 0 18 Apr 2025
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion Longguang Zhong Fanqi Wan Ziyi Yang Guosheng Liang Tianyuan Shi Xiaojun Quan MoMe 57 0 0 09 Apr 2025
PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learning Xinpeng Ding Kaipeng Zhang Jinahua Han Lanqing Hong Hang Xu Xiaomeng Li MLLM VLM 178 0 0 08 Apr 2025
Information-Theoretic Reward Decomposition for Generalizable RLHF Liyuan Mao Haoran Xu Amy Zhang Weinan Zhang Chenjia Bai 33 0 0 08 Apr 2025
SEA-LION: Southeast Asian Languages in One Network Raymond Ng Thanh Ngan Nguyen Yuli Huang Ngee Chia Tai Wai Yi Leong ... David Ong Tat-Wee B. Liu William-Chandra Tjhi Erik Cambria Leslie Teo 36 11 0 08 Apr 2025
Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use Anna Goldie Azalia Mirhoseini Hao Zhou Irene Cai Christopher D. Manning SyDa OffRL ReLM LRM 109 3 0 07 Apr 2025
CARE: Aligning Language Models for Regional Cultural Awareness Geyang Guo Tarek Naous Hiromi Wakaki Yukiko Nishimura Yuki Mitsufuji Alan Ritter Wei-ping Xu 52 0 0 07 Apr 2025
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning Kai Ye Hongyi Zhou Jin Zhu Francesco Quinzan C. Shi 30 1 0 03 Apr 2025
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models Yang Sui Yu-Neng Chuang Guanchu Wang Jiamu Zhang Tianyi Zhang ... Hongyi Liu Andrew Wen Shaochen Zhong Hanjie Chen OffRL ReLM LRM 75 29 0 20 Mar 2025
From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment J. Li Jian-Yu Guan Songhao Wu Wei Yu Wu Rui Yan 62 1 0 19 Mar 2025
Aligning Multimodal LLM with Human Preference: A Survey Tao Yu Yuhui Zhang Chaoyou Fu Junkang Wu Jinda Lu ... Qingsong Wen Z. Zhang Yan Huang Liang Wang T. Tan 164 2 0 18 Mar 2025
Fine-Tuning Diffusion Generative Models via Rich Preference Optimization Hanyang Zhao Haoxian Chen Yucheng Guo Genta Indra Winata Tingting Ou Ziyu Huang D. Yao Wenpin Tang 59 0 0 13 Mar 2025
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning Bowen Jin Hansi Zeng Zhenrui Yue Dong Wang Sercan Ö. Arik Dong Wang Hamed Zamani J. Han RALM ReLM KELM OffRL AI4TS LRM 84 24 0 12 Mar 2025
Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis Letian Zhang Quan Cui Bingchen Zhao Cheng Yang MLLM SyDa 54 0 0 11 Mar 2025
SOLAR: Scalable Optimization of Large-scale Architecture for Reasoning Chen Li Yinyi Luo Anudeep Bolimera Uzair Ahmed Shuifa Sun Hrishikesh Gokhale Marios Savvides LRM AI4CE 60 1 0 06 Mar 2025
DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models Ruizhe Chen Wenhao Chai Zhifei Yang Xiaotian Zhang Qiufeng Wang Tony Q. S. Quek Soujanya Poria Zuozhu Liu 50 0 0 06 Mar 2025
DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models Yi Shen Jing Zhang Jieyun Huang Shuming Shi Wenjing Zhang Jiangze Yan Jiaojiao Zhao Kai Wang Shiguo Lian LRM 80 15 0 06 Mar 2025
LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning Zhibin Lan Liqiang Niu Fandong Meng Jie Zhou Jinsong Su VLM 69 1 0 04 Mar 2025
PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation Yuxuan Liu 45 0 0 03 Mar 2025
Plan2Align: Predictive Planning Based Test-Time Preference Alignment in Paragraph-Level Machine Translation Kuang-Da Wang Teng-Ruei Chen Yu-Heng Hung Shuoyang Ding Yueh-Hua Wu Yu-Chun Wang Chao-Han Huck Yang Wen-Chih Peng Ping-Chun Hsieh 74 0 0 28 Feb 2025
Kanana: Compute-efficient Bilingual Language Models Kanana LLM Team Yunju Bak Hojin Lee Minho Ryu Jiyeon Ham ... Daniel Lee Minchul Lee M. Lee Shinbok Lee Gaeun Seo 90 1 0 26 Feb 2025
Advantage-Guided Distillation for Preference Alignment in Small Language Models Shiping Gao Fanqi Wan Jiajian Guo Xiaojun Quan Qifan Wang ALM 58 0 0 25 Feb 2025
Stackelberg Game Preference Optimization for Data-Efficient Alignment of Language Models Xu Chu Zhixin Zhang Tianyu Jia Yujie Jin 77 0 0 25 Feb 2025
Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Support G. Wang Minyu Gao Shuai Yang Ya Zhang Lizhi He ... Yexuan Zhang Wanyue Li Lu Chen Jintao Fei Xin Li 113 1 0 25 Feb 2025
LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjoint Qianli Ma Dongrui Liu Qian Chen Linfeng Zhang Jing Shao MoMe 144 0 0 24 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Z. Chen Mingxiao Li Shangsong Liang Z. Ren V. Honavar 95 5 0 21 Feb 2025
BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment Sizhe Wang Yongqi Tong Hengyuan Zhang Dawei Li Xin Zhang Tianlong Chen 85 5 0 21 Feb 2025
Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF Shicong Cen Jincheng Mei Katayoon Goshvadi Hanjun Dai Tong Yang Sherry Yang Dale Schuurmans Yuejie Chi Bo Dai OffRL 65 23 0 20 Feb 2025
Faster WIND: Accelerating Iterative Best-of- $N$ Distillation for LLM Alignment Tong Yang Jincheng Mei H. Dai Zixin Wen Shicong Cen Dale Schuurmans Yuejie Chi Bo Dai 45 4 0 20 Feb 2025
Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees Yongtao Wu Luca Viano Yihang Chen Zhenyu Zhu Kimon Antonakopoulos Quanquan Gu V. Cevher 54 0 0 18 Feb 2025