Reinforcement learning (RL) is vital for optimizing large language models (LLMs). Recent Group Relative Policy Optimization (GRPO) estimates advantages using multiple on-policy outputs per prompt, leading to high computational costs and low data efficiency. To address this, we introduce Replay-Enhanced Policy Optimization (RePO), which leverages diverse replay strategies to retrieve off-policy samples from a replay buffer, allowing policy optimization based on a broader and more diverse set of samples for each prompt. Experiments on five LLMs across seven mathematical reasoning benchmarks demonstrate that RePO achieves absolute average performance gains of and points for Qwen2.5-Math-1.5B and Qwen3-1.7B, respectively, compared to GRPO. Further analysis indicates that RePO increases computational cost by while raising the number of effective optimization steps by for Qwen3-1.7B, with both on-policy and off-policy sample numbers set to . The repository can be accessed atthis https URL.
View on arXiv@article{li2025_2506.09340,
title={ RePO: Replay-Enhanced Policy Optimization },
author={ Siheng Li and Zhanhui Zhou and Wai Lam and Chao Yang and Chaochao Lu },
journal={arXiv preprint arXiv:2506.09340},
year={ 2025 }
}