Retrospective Policy Gradient

Policy Gradient (PG) methods represent a notable category of reinforcement learning algorithms, particularly effective in continuous action domains. These techniques adjust the parameters of parametric policies through stochastic gradient ascent, typically utilizing on-policy trajectory samples to estimate the policy gradient. Nevertheless, this reliance on newly collected data makes them sample-inefficient. Specifically, standard PG algorithms require O(1/ε²) trajectories to achieve an ε-approximate stationary point. A widely used strategy to improve efficiency involves leveraging off-policy information from previous iterations, such as earlier gradient estimates or trajectories. Although the theoretical advantages of gradient reuse are well established—leading to improved convergence rates of O(1/ε^(3/2))—the theoretical understanding of reusing past trajectories remains relatively limited. We propose a power-mean correction for the multiple importance weighting estimator and introduce the Retrospective Policy Gradient (RPG), a PG algorithm that integrates both past and current trajectories for policy updates. Our implementation builds upon standard RL frameworks and supports both linear and deep neural network policies. We evaluate RPG on continuous control benchmarks from the MuJoCo Gym suite (Half-Cheetah, Swimmer) and custom environments (Cart-Pole). We compare against REINFORCE, GPOMDP, SVRPG, SRVRPG, STORM-PG, and DEF-PG. Empirical results demonstrate that RPG consistently outperforms existing policy gradient algorithms in terms of both sample efficiency and final empirical performance. In particular, it achieves higher cumulative rewards using significantly fewer environment interactions. We also provide sensitivity analyses with respect to the reuse buffer size. Through a novel theoretical analysis, we show that the Retrospective Policy Gradient achieves a sample complexity of O(1/ε), marking the best-known rate to date. Empirical findings further validate the effectiveness of our method compared to state-of-the-art PG algorithms.

Author dc:creator

• Federico Mansutti (23292028)

• Reinforcement Learning
• Computer Science

Statement dc:rights

• In Copyright
• Open Access after 2028-01-01

DOI dc:identifier

https://doi.org/10.25417/uic.31451704.v1

OAI identifier oai:identifier

oai:figshare.com:article/31451704