BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment

21 February 2025

Papers citing "BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment"

37 / 37 papers shown

Title
The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation Ruichen Zhang Rana Muhammad Shahroz Khan Zhen Tan Dawei Li Song Wang Tianlong Chen LRM 43 0 0 24 May 2025
Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundary Licheng Pan Yongqi Tong Xin Zhang Xiaolu Zhang Jun Zhou Zhixuan Chu 43 0 0 23 May 2025
Quantifying the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Data Shiping Yang Jie Wu Wenbiao Ding Ning Wu Shining Liang Ming Gong Hengyuan Zhang Dongmei Zhang AAML 153 7 0 07 Mar 2025
Larger or Smaller Reward Margins to Select Preferences for Alignment? Kexin Huang Junkang Wu Ziqian Chen Xue Wang Jinyang Gao Bolin Ding Jiancan Wu Xiangnan He Xiang Wang 73 1 0 25 Feb 2025
Preference Leakage: A Contamination Problem in LLM-as-a-judge Dawei Li Renliang Sun Yue Huang Ming Zhong Bohan Jiang Jiawei Han Wei Wei Wei Wang Huan Liu 155 30 0 03 Feb 2025
Assessing the Impact of Conspiracy Theories Using Large Language Models Bohan Jiang Dawei Li Zhen Tan Xinyi Zhou Ashwin Rao Kristina Lerman H. Bernard Huan Liu 189 2 0 09 Dec 2024
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge Dawei Li Bohan Jiang Liangjie Huang Alimohammad Beigi Chengshuai Zhao ... Canyu Chen Tianhao Wu Kai Shu Lu Cheng Huan Liu ELM AILaw 318 112 0 25 Nov 2024
Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning Tianduo Wang Shichen Li Wei Lu LRM AI4CE 78 20 1 25 Jul 2024
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline Tianle Li Wei-Lin Chiang Evan Frick Lisa Dunlap Tianhao Wu Banghua Zhu Joseph E. Gonzalez Ion Stoica ALM 97 182 0 17 Jun 2024
SimPO: Simple Preference Optimization with a Reference-Free Reward Yu Meng Mengzhou Xia Danqi Chen 145 492 0 23 May 2024
DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature Dawei Li Shu Yang Zhen Tan Jae Young Baik Sunkwon Yun ... D. Duong-Tran Ying Ding Huan Liu Li Shen Tianlong Chen 94 40 0 08 May 2024
Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks Amir Saeidi Shivanshu Verma Chitta Baral Chitta Baral ALM 98 26 0 23 Apr 2024
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators Yann Dubois Balázs Galambosi Percy Liang Tatsunori Hashimoto ALM 146 402 0 06 Apr 2024
Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment Feifan Song Bowen Yu Hao Lang Haiyang Yu Fei Huang Houfeng Wang Yongbin Li ALM 75 15 0 17 Mar 2024
Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences Pulkit Pattnaik Rishabh Maheshwary Kelechi Ogueji Vikas Yadav Sathwik Tejaswi Madhusudhan 61 21 0 12 Mar 2024
Large Language Models for Data Annotation: A Survey Zhen Tan Dawei Li Song Wang Alimohammad Beigi Bohan Jiang Amrita Bhattacharjee Mansooreh Karami Wenlin Yao Lu Cheng Huan Liu SyDa 121 79 0 21 Feb 2024
LESS: Selecting Influential Data for Targeted Instruction Tuning Mengzhou Xia Sadhika Malladi Suchin Gururangan Sanjeev Arora Danqi Chen 155 245 0 06 Feb 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 283 569 0 02 Feb 2024
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning Wei Liu Weihao Zeng Keqing He Yong Jiang Junxian He ALM 106 239 0 25 Dec 2023
Safe RLHF: Safe Reinforcement Learning from Human Feedback Josef Dai Xuehai Pan Ruiyang Sun Jiaming Ji Xinbo Xu Mickel Liu Yizhou Wang Yaodong Yang 131 364 0 19 Oct 2023
Statistical Rejection Sampling Improves Preference Optimization Tianqi Liu Yao-Min Zhao Rishabh Joshi Misha Khalman Mohammad Saleh Peter J. Liu Jialu Liu 133 249 0 13 Sep 2023
InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4 Lai Wei Zihao Jiang Weiran Huang Lichao Sun VLM MLLM 97 60 0 23 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 419 12,076 0 18 Jul 2023
Understanding In-Context Learning via Supportive Pretraining Data Xiaochuang Han Daniel Simig Todor Mihaylov Yulia Tsvetkov Asli Celikyilmaz Tianlu Wang AIMat 106 38 0 26 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 458 4,444 0 09 Jun 2023
Benchmarking Foundation Models with Language-Model-as-an-Examiner Yushi Bai Jiahao Ying Yixin Cao Xin Lv Yuze He ... Yijia Xiao Haozhe Lyu Jiayin Zhang Juanzi Li Lei Hou ALM ELM 103 149 0 07 Jun 2023
Fine-Tuning Language Models with Advantage-Induced Policy Alignment Banghua Zhu Hiteshi Sharma Felipe Vieira Frujeri Shi Dong Chenguang Zhu Michael I. Jordan Jiantao Jiao OSLM 52 41 0 04 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 389 4,169 0 29 May 2023
LIMA: Less Is More for Alignment Chunting Zhou Pengfei Liu Puxin Xu Srini Iyer Jiao Sun ... Susan Zhang Gargi Ghosh M. Lewis Luke Zettlemoyer Omer Levy ALM 115 853 0 18 May 2023
Human-like Summarization Evaluation with ChatGPT Mingqi Gao Jie Ruan Renliang Sun Xunjian Yin Shiping Yang Xiaojun Wan ALM AI4MH 61 135 0 05 Apr 2023
Calibrating Sequence likelihood Improves Conditional Language Generation Yao-Min Zhao Misha Khalman Rishabh Joshi Shashi Narayan Mohammad Saleh Peter J. Liu UQLM 102 135 0 30 Sep 2022
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen ... Jack Clark Sam McCandlish C. Olah Benjamin Mann Jared Kaplan 256 2,627 0 12 Apr 2022
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 502 10,526 0 17 Jun 2021
Estimating Training Data Influence by Tracing Gradient Descent G. Pruthi Frederick Liu Mukund Sundararajan Satyen Kale TDI 112 419 0 19 Feb 2020
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 583 19,315 0 20 Jul 2017
Deep reinforcement learning from human preferences Paul Christiano Jan Leike Tom B. Brown Miljan Martic Shane Legg Dario Amodei 218 3,377 0 12 Jun 2017
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.1K 150,433 0 22 Dec 2014