Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

50 / 2,637 papers shown

Title
Preference Optimization for Reasoning with Pseudo Feedback Fangkai Jiao Geyang Guo Xingxing Zhang Nancy F. Chen Chenyu You Furu Wei LRM 111 9 0 17 Feb 2025
Unhackable Temporal Rewarding for Scalable Video MLLMs En Yu Kangheng Lin Liang Zhao Yana Wei Zining Zhu ... Jianjian Sun Zheng Ge Xinsong Zhang Jingyu Wang Wenbing Tao 69 4 0 17 Feb 2025
Scaling Autonomous Agents via Automatic Reward Modeling And Planning Zhenfang Chen Delin Chen Rui Sun Wenjun Liu Chuang Gan LLMAG 64 3 0 17 Feb 2025
Which Retain Set Matters for LLM Unlearning? A Case Study on Entity Unlearning Hwan Chang Hwanhee Lee MU 49 0 0 17 Feb 2025
LeDex: Training LLMs to Better Self-Debug and Explain Code Nan Jiang Xiaopeng Li Shiqi Wang Qiang Zhou Soneya Binta Hossain Baishakhi Ray Varun Kumar Xiaofei Ma Anoop Deoras LRM 92 12 0 17 Feb 2025
Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models Yingshui Tan Yilei Jiang Heng Chang Jiaheng Liu Xingyuan Bu Wenbo Su Xiangyu Yue Xiaoyong Zhu Bo Zheng ALM 93 1 0 17 Feb 2025
HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation L. Yang Xinchen Zhang Ye Tian Chenming Shang Minghao Xu Wentao Zhang Bin Cui 102 1 0 17 Feb 2025
Prompt Inject Detection with Generative Explanation as an Investigative Tool Jonathan Pan Swee Liang Wong Yidi Yuan Xin Wei Chia SILM 61 0 0 16 Feb 2025
Leveraging Constrained Monte Carlo Tree Search to Generate Reliable Long Chain-of-Thought for Mathematical Reasoning Qingwen Lin Boyan Xu Zijian Li Zhifeng Hao Keli Zhang Ruichu Cai LRM 57 3 0 16 Feb 2025
To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models Zihao Zhu Hongbao Zhang Ruotong Wang Ke Xu Siwei Lyu Baoyuan Wu AAML LRM 69 5 0 16 Feb 2025
ReLearn: Unlearning via Learning for Large Language Models Haoming Xu Ningyuan Zhao Liming Yang Sendong Zhao Shumin Deng Mengru Wang Bryan Hooi Nay Oo Huajun Chen N. Zhang KELM CLL MU 273 0 0 16 Feb 2025
Maximize Your Diffusion: A Study into Reward Maximization and Alignment for Diffusion-based Control Dom Huh P. Mohapatra 94 1 0 16 Feb 2025
PlanGenLLMs: A Modern Survey of LLM Planning Capabilities Hui Wei Zihao Zhang Shenghua He Tian Xia Shijia Pan Fei Liu 71 6 0 16 Feb 2025
LogiDynamics: Unraveling the Dynamics of Logical Inference in Large Language Model Reasoning Tianshi Zheng Jiayang Cheng Chunyang Li Haochen Shi Junyao Xing Jiaxin Bai Yangqiu Song Ginny Wong Simon See LRM 51 3 0 16 Feb 2025
Preference learning made easy: Everything should be understood through win rate Lily H. Zhang Rajesh Ranganath 87 0 0 14 Feb 2025
Efficient Multitask Learning in Small Language Models Through Upside-Down Reinforcement Learning Yu-Chen Lin Sanat Sharma Hari Manikandan Jayant Kumar Tracy Holloway King Jing Zheng LRM 68 1 0 14 Feb 2025
Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generation Seungjun Yu Kisung Kim Daejung Kim Haewook Han Jinhan Lee 77 0 0 14 Feb 2025
Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model Guoqing Ma Haoyang Huang K. Yan L. Chen Nan Duan ... Yansen Wang Yuanwei Lu Yu-Cheng Chen Yu-Juan Luo Yihao Luo DiffM VGen 182 19 0 14 Feb 2025
KKA: Improving Vision Anomaly Detection through Anomaly-related Knowledge from Large Language Models Dong Chen Zhengqing Hu Peiguang Fan Yueting Zhuang Yafei Li Qidong Liu Xiaoheng Jiang Mingliang Xu VLM 55 0 0 14 Feb 2025
Large Language Diffusion Models Shen Nie Fengqi Zhu Zebin You Xiaolu Zhang Jingyang Ou Jun Hu Jun Zhou Yankai Lin Ji-Rong Wen Chongxuan Li 117 22 0 14 Feb 2025
Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models Xin Zhou Yiwen Guo Ruotian Ma Tao Gui Qi Zhang Xuanjing Huang LRM 92 2 0 13 Feb 2025
RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation C. Zhou Xinyu Zhang Dandan Song Xiancai Chen Wanli Gu Huipeng Ma Yuhang Tian Hao Fei Linmei Hu 63 1 0 13 Feb 2025
Typhoon T1: An Open Thai Reasoning Model Pittawat Taveekitworachai Potsawee Manakul Kasima Tharnpipitchai Kunat Pipatanakul OffRL LRM 110 0 0 13 Feb 2025
DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimization Xuefeng Liu Songhao Jiang Siyu Chen Zhuoran Yang Yuxin Chen Ian Foster Rick L. Stevens LM&MA OffRL 63 0 0 11 Feb 2025
Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates Jincheng Mei Bo Dai Alekh Agarwal Sharan Vaswani Anant Raj Csaba Szepesvári Dale Schuurmans 89 0 0 11 Feb 2025
Trustworthy AI on Safety, Bias, and Privacy: A Survey Xingli Fang Jianwei Li Varun Mulchandani Jung-Eun Kim 50 0 0 11 Feb 2025
LUNAR: LLM Unlearning via Neural Activation Redirection William F. Shen Xinchi Qiu Meghdad Kurmanji Alex Iacob Lorenzo Sani Yihong Chen Nicola Cancedda Nicholas D. Lane MU 59 1 0 11 Feb 2025
Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM Qingshui Gu Shu Li Tianyu Zheng Zhaoxiang Zhang 307 0 0 10 Feb 2025
Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection Yan Weng Fengbin Zhu Tong Ye Haoyan Liu Fuli Feng Tat-Seng Chua RALM 110 1 0 10 Feb 2025
ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates L. Yang Zhaochen Yu Bin Cui Mengdi Wang ReLM LRM AI4CE 101 12 0 10 Feb 2025
AI Alignment at Your Discretion Maarten Buyl Hadi Khalaf C. M. Verdun Lucas Monteiro Paes Caio Vieira Machado Flavio du Pin Calmon 48 0 0 10 Feb 2025
Jailbreaking to Jailbreak Jeremy Kritz Vaughn Robinson Robert Vacareanu Bijan Varjavand Michael Choi Bobby Gogov Scale Red Team Summer Yue Willow Primack Zifan Wang 258 2 0 09 Feb 2025
Learning to Substitute Words with Model-based Score Ranking Hongye Liu Ricardo Henao 48 0 0 09 Feb 2025
PIPA: Preference Alignment as Prior-Informed Statistical Estimation Junbo Li Zhangyang Wang Qiang Liu OffRL 108 0 0 09 Feb 2025
Self-Training Large Language Models for Tool-Use Without Demonstrations Ne Luo Aryo Pradipta Gema Xuanli He Emile van Krieken Pietro Lesci Pasquale Minervini LLMAG 87 1 0 09 Feb 2025
Nearly Optimal Sample Complexity of Offline KL-Regularized Contextual Bandits under Single-Policy Concentrability Qingyue Zhao Kaixuan Ji Heyang Zhao Tong Zhang Q. Gu OffRL 50 0 0 09 Feb 2025
Dual Caption Preference Optimization for Diffusion Models Amir Saeidi Yiran Luo Agneet Chatterjee Shamanthak Hegde Bimsara Pathiraja Yezhou Yang Chitta Baral DiffM 65 0 0 09 Feb 2025
Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions Jingxin Xu Guoshun Nan Sheng Guan Sicong Leng Yang Liu Zixiao Wang Yuyang Ma Zhili Zhou Yanzhao Hou Xiaofeng Tao LM&MA 60 0 0 08 Feb 2025
Design Considerations in Offline Preference-based RL Alekh Agarwal Christoph Dann T. V. Marinov OffRL 61 0 0 08 Feb 2025
Evolving LLMs' Self-Refinement Capability via Iterative Preference Optimization Yongcheng Zeng Xinyu Cui Xuanfa Jin Guoqing Liu Zexu Sun Quan He Dong Li Ning Yang Haifeng Zhang Jun Wang LLMAG LRM 100 1 0 08 Feb 2025
Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet Berk Atil Vipul Gupta Sarkar Snigdha Sarathi Das R. Passonneau 255 0 0 07 Feb 2025
Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance Shehzeen Samarah Hussain Paarth Neekhara Xuesong Yang Edresson Casanova Subhankar Ghosh Mikyas T. Desta Roy Fejgin Rafael Valle Jason Chun Lok Li 69 3 0 07 Feb 2025
LLMs Can Teach Themselves to Better Predict the Future Benjamin Turtel Danny Franklin Philipp Schoenegger LRM 73 0 0 07 Feb 2025
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang Qirun Dai Hao Peng ALM 120 4 0 06 Feb 2025
Recommendations Beyond Catalogs: Diffusion Models for Personalized Generation Gabriel Patron Zhiwei Xu Ishan Kapnadak Felipe Maia Polo DiffM 43 0 0 05 Feb 2025
CTR-Driven Advertising Image Generation with Multimodal Large Language Models Xingye Chen Wei Feng Zhenbang Du Weizhen Wang Yuxiao Chen ... Jingping Shao Yuanjie Shao Xinge You Changxin Gao Nong Sang OffRL 52 2 0 05 Feb 2025
Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge Daniel Tamayo Aitor Gonzalez-Agirre Javier Hernando Marta Villegas KELM 101 4 0 04 Feb 2025
Generative Psycho-Lexical Approach for Constructing Value Systems in Large Language Models Haoran Ye Tianze Zhang Yuhang Xie Liyuan Zhang Yuanyi Ren Xin Zhang Guojie Song PILM 88 0 0 04 Feb 2025
QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search Zongyu Lin Yao Tang Xingcheng Yao Da Yin Ziniu Hu Ningyu Zhang Kai-Wei Chang LRM 50 4 0 04 Feb 2025
Rethinking Homogeneity of Vision and Text Tokens in Large Vision-and-Language Models Chia-Wen Kuo Sijie Zhu Fan Chen Xiaohui Shen Longyin Wen VLM 65 1 0 04 Feb 2025