What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret

3 March 2025

Papers citing "What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret"

16 / 16 papers shown

Title
PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier Y. Jiang Yuwen Xiong Yufeng Yuan Chao Xin Wenyuan Xu Yu Yue Qianchuan Zhao Lin Yan LRM 122 0 0 12 Jun 2025
Diversity-Aware Policy Optimization for Large Language Model Reasoning Jian Yao Ran Cheng Xingyu Wu Jibin Wu Kay Chen Tan LRM 99 0 0 29 May 2025
Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles Jiangjie Chen Qianyu He Siyu Yuan Aili Chen Zhicheng Cai ... Qiying Yu Xuefeng Li Jiaze Chen Hao Zhou Mingxuan Wang ReLM LRM 105 2 0 26 May 2025
Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models Haoyuan Sun Jiaqi Wu Bo Xia Yifu Luo Yifei Zhao Kai Qin Xufei Lv Tiantian Zhang Yongzhe Chang Xueqian Wang OffRL LRM 209 0 0 24 May 2025
QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning Fanqi Wan Weizhou Shen Shengyi Liao Yingcheng Shi Chenliang Li Ziyi Yang Ji Zhang Fei Huang Jingren Zhou Ming Yan OffRL LLMAG ReLM LRM 110 0 0 23 May 2025
Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning Yurun Yuan Fan Chen Zeyu Jia Alexander Rakhlin Tengyang Xie OffRL 135 1 0 21 May 2025
Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment Siliang Zeng Quan Wei William Brown Oana Frunza Yuriy Nevmyvaka Mingyi Hong LRM 113 2 0 17 May 2025
Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL Songjun Tu Jiahao Lin Qichao Zhang Xiangyu Tian Linjing Li Xiangyuan Lan Dongbin Zhao OffRL ReLM LRM 101 2 0 16 May 2025
Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models Guanghao Zhou Panjia Qiu Chong Chen Jiadong Wang Zheming Yang Jian Xu Minghui Qiu OffRL LRM 212 8 0 30 Apr 2025
Reinforcement Learning for Reasoning in Large Language Models with One Training Example Yiping Wang Qing Yang Zhiyuan Zeng Liliang Ren Liu Liu ... Jianfeng Gao Weizhu Chen Shuaiqiang Wang Simon Shaolei Du Yelong Shen OffRL ReLM LRM 347 47 0 29 Apr 2025
Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning Yixuan Even Xu Yash Savani Fei Fang Zico Kolter OffRL 115 12 0 18 Apr 2025
Concise Reasoning via Reinforcement Learning Mehdi Fatemi Banafsheh Rafiee Mingjie Tang Kartik Talamadupula ReLM OffRL LRM 143 17 0 07 Apr 2025
VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks Yu Yue Yufeng Yuan Qiying Yu Xiaochen Zuo Ruofei Zhu ... Ru Zhang Xin Liu Mingxuan Wang Yonghui Wu Lin Yan OffRL LRM 143 39 0 07 Apr 2025
A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond Xiaoye Qu Yafu Li Zhaochen Su Weigao Sun Jianhao Yan ... Chaochao Lu Yue Zhang Xian-Sheng Hua Bowen Zhou Yu Cheng ReLM OffRL LRM 189 47 0 27 Mar 2025
DAPO: An Open-Source LLM Reinforcement Learning System at Scale Qiying Yu Zheng Zhang Ruofei Zhu Yufeng Yuan Xiaochen Zuo ... Ya Zhang Lin Yan Mu Qiao Yonghui Wu Mingxuan Wang OffRL LRM 248 217 0 18 Mar 2025
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI Daya Guo Dejian Yang Haowei Zhang Junxiao Song ... Shiyu Wang S. Yu Shunfeng Zhou Shuting Pan S.S. Li ReLM VLM OffRL AI4TS LRM 395 2,033 0 22 Jan 2025