Direct Preference Optimization: Your Language Model is Secretly a Reward Model

29 May 2023

Christopher D. Manning

Chelsea Finn

ALM

ArXiv PDF HTML

Papers citing "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"

34 / 2,684 papers shown

Title
Detecting and Preventing Hallucinations in Large Vision Language Models Anisha Gunjal Jihan Yin Erhan Bas MLLM VLM 36 158 0 11 Aug 2023
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization Weiran Yao Shelby Heinecke Juan Carlos Niebles Zhiwei Liu Yihao Feng ... Ran Xu P. Mùi Haiquan Wang Caiming Xiong Silvio Savarese LLMAG LM&Ro 41 76 0 04 Aug 2023
Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges Giorgio Franceschelli Mirco Musolesi AI4CE 56 21 0 31 Jul 2023
Scaling Sentence Embeddings with Large Language Models Ting Jiang Shaohan Huang Zhongzhi Luan Deqing Wang Fuzhen Zhuang LRM 49 41 0 31 Jul 2023
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback Stephen Casper Xander Davies Claudia Shi T. Gilbert Jérémy Scheurer ... Erdem Biyik Anca Dragan David M. Krueger Dorsa Sadigh Dylan Hadfield-Menell ALM OffRL 52 487 0 27 Jul 2023
Leveraging Implicit Feedback from Deployment Data in Dialogue Richard Yuanzhe Pang Stephen Roller Kyunghyun Cho He He Jason Weston 59 8 0 26 Jul 2023
RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment Kevin Kaichuang Yang Dan Klein Asli Celikyilmaz Nanyun Peng Yuandong Tian ALM 51 32 0 24 Jul 2023
A Comprehensive Overview of Large Language Models Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Naveed Akhtar Nick Barnes Ajmal Mian OffRL 76 561 0 12 Jul 2023
TIM: Teaching Large Language Models to Translate with Comparison Jiali Zeng Fandong Meng Yongjing Yin Jie Zhou 42 55 0 10 Jul 2023
Let Me Teach You: Pedagogical Foundations of Feedback for Language Models Beatriz Borges Niket Tandon Tanja Käser Antoine Bosselut 55 4 0 01 Jul 2023
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting Zhen Qin R. Jagerman Kai Hui Honglei Zhuang Junru Wu ... Tianqi Liu Jialu Liu Donald Metzler Xuanhui Wang Michael Bendersky ALM RALM 58 230 0 30 Jun 2023
Preference Ranking Optimization for Human Alignment Feifan Song Yu Bowen Minghao Li Haiyang Yu Fei Huang Yongbin Li Houfeng Wang ALM 34 243 0 30 Jun 2023
A Survey on Multimodal Large Language Models Shukang Yin Chaoyou Fu Sirui Zhao Ke Li Xing Sun Tong Xu Enhong Chen MLLM LRM 64 565 0 23 Jun 2023
AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks Alexander Tornede Difan Deng Theresa Eimer Joseph Giovanelli Aditya Mohan ... Sarah Segel Daphne Theodorakopoulos Tanja Tornede Henning Wachsmuth Marius Lindauer 43 23 0 13 Jun 2023
Large Language Models Sometimes Generate Purely Negatively-Reinforced Text Fabien Roger SILM 34 0 0 13 Jun 2023
Artificial General Intelligence for Medical Imaging Xiang Li Lu Zhang Zihao Wu Zheng Liu Lin Zhao ... Pingkuan Yan Quanzheng Li Wen Liu Tianming Liu Dinggang Shen LM&MA AI4CE 69 40 0 08 Jun 2023
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming Hussein Mozannar Gagan Bansal Adam Fourney Eric Horvitz 57 26 0 08 Jun 2023
Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs Alexander K. Lew Tan Zhi-Xuan Gabriel Grand Vikash K. Mansinghka LLMSV LRM 67 35 0 05 Jun 2023
Fine-Tuning Language Models with Advantage-Induced Policy Alignment Banghua Zhu Hiteshi Sharma Felipe Vieira Frujeri Shi Dong Chenguang Zhu Michael I. Jordan Jiantao Jiao OSLM 36 39 0 04 Jun 2023
Training Socially Aligned Language Models on Simulated Social Interactions Ruibo Liu Ruixin Yang Chenyan Jia Ge Zhang Denny Zhou Andrew M. Dai Diyi Yang Soroush Vosoughi ALM 42 47 0 26 May 2023
Inverse Preference Learning: Preference-based RL without a Reward Function Joey Hejna Dorsa Sadigh OffRL 51 49 0 24 May 2023
Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models Ashutosh Baheti Ximing Lu Faeze Brahman Ronan Le Bras Maarten Sap Mark O. Riedl 43 9 0 24 May 2023
On Learning to Summarize with Large Language Models as References Yixin Liu Kejian Shi Katherine S He Longtian Ye Alexander R. Fabbri Pengfei Liu Dragomir R. Radev Arman Cohan ELM 50 74 0 23 May 2023
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback Yann Dubois Xuechen Li Rohan Taori Tianyi Zhang Ishaan Gulrajani Jimmy Ba Carlos Guestrin Percy Liang Tatsunori B. Hashimoto ALM 53 562 0 22 May 2023
Diffusion Language Models Generation Can Be Halted Early Sofia Maria Lo Cicero Vaina Nikita Balagansky Daniil Gavrilov DiffM 64 0 0 18 May 2023
Consistency Regularization for Domain Generalization with Logit Attribution Matching Han Gao Kaican Li Weiyan Xie Zhi Lin Yongxiang Huang Luning Wang Caleb Chen Cao N. Zhang 37 2 0 13 May 2023
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation Patrick Fernandes Aman Madaan Emmy Liu António Farinhas Pedro Henrique Martins ... José G. C. de Souza Shuyan Zhou Tongshuang Wu Graham Neubig André F. T. Martins ALM 123 56 0 01 May 2023
On the Creativity of Large Language Models Giorgio Franceschelli Mirco Musolesi 111 55 0 27 Mar 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 445 2,232 0 22 Mar 2023
Direct Preference-based Policy Optimization without Reward Modeling Gaon An Junhyeok Lee Xingdong Zuo Norio Kosaka KyungHyun Kim Hyun Oh Song OffRL 37 28 0 30 Jan 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 459 12,359 0 04 Mar 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 256 1,669 0 15 Oct 2021
Creativity and Machine Learning: A Survey Giorgio Franceschelli Mirco Musolesi VLM AI4CE 39 40 0 06 Apr 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 310 1,646 0 18 Sep 2019