What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret

3 March 2025

Papers citing "What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret"

7 / 7 papers shown

Title
Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL Songjun Tu Jiahao Lin Qichao Zhang Xiangyu Tian Linjing Li Xiangyuan Lan Dongbin Zhao OffRL ReLM LRM 21 0 0 16 May 2025
Reinforcement Learning for Reasoning in Large Language Models with One Training Example Yiping Wang Qing Yang Zhiyuan Zeng Liliang Ren L. Liu ... Jianfeng Gao Weizhu Chen S. Wang Simon S. Du Yelong Shen OffRL ReLM LRM 120 5 0 29 Apr 2025
Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning Yixuan Even Xu Yash Savani Fei Fang Zico Kolter OffRL 42 2 0 18 Apr 2025
VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks Yu Yue Yufeng Yuan Qiying Yu Xiaochen Zuo Ruofei Zhu ... Ru Zhang Xin Liu Mingxuan Wang Yonghui Wu Lin Yan OffRL LRM 29 8 0 07 Apr 2025
Concise Reasoning via Reinforcement Learning Mehdi Fatemi Banafsheh Rafiee Mingjie Tang Kartik Talamadupula ReLM OffRL LRM 52 6 0 07 Apr 2025
A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond Xiaoye Qu Yafu Li Zhaochen Su Weigao Sun Jianhao Yan ... Chaochao Lu Yue Zhang Xian-Sheng Hua Bowen Zhou Yu Cheng ReLM OffRL LRM 88 16 0 27 Mar 2025
DAPO: An Open-Source LLM Reinforcement Learning System at Scale Qiying Yu Zhe Zhang Ruofei Zhu Yufeng Yuan Xiaochen Zuo ... Ya-Qin Zhang Lin Yan Mu Qiao Yonghui Wu Mingxuan Wang OffRL LRM 67 54 0 18 Mar 2025