One-sentence Summary

Researchers at the University of California, Berkeley propose Trust Region Policy Optimization (TRPO), a practical algorithm that approximates a theoretically-justified scheme for guaranteed monotonic improvement when optimizing large nonlinear policies such as neural networks, demonstrating robust performance on robotic locomotion tasks and Atari games with minimal hyperparameter tuning.

Key Contributions

• Introduces Trust Region Policy Optimization (TRPO), a practical algorithm for optimizing large nonlinear stochastic policies, including neural-network policies, by updating policies under a KL divergence trust region constraint while optimizing a local surrogate to expected return.

• Analyzes KL-divergence-penalized policy updates and shows that the trust-region approach can guarantee monotonic policy improvement; the analysis also frames policy gradient and policy iteration methods as limiting cases of optimizing an objective under a trust region constraint.

• Demonstrates strong empirical results on simulated robotic swimming, hopping, and walking, and on Atari gameplay using raw screen images, with little hyperparameter tuning and observed monotonic improvement.

Introduction

Policy optimization is central to reinforcement learning for control, robotics, and game playing, where the goal is to improve large, expressive policies such as neural networks. Prior methods include policy iteration, policy gradient methods, and derivative-free search, but gradient based approaches often underperform random search in practice despite better theoretical sample complexity, and naive updates can be unstable because policy changes alter the state visitation distribution in complex ways. The authors introduce trust region policy optimization, a scalable algorithm that improves a surrogate advantage objective while constraining policy updates with a KL divergence trust region, yielding a monotonic improvement guarantee and practical single-path and vine variants. The method trains large nonlinear policies and successfully learns locomotion and Atari control tasks, while unifying policy gradient and policy iteration under a common trust region framework.

Method

Method architecture and optimization pipeline

The authors leverage a trust region framework for stochastic policy optimization. The core idea is to replace the original policy performance objective with a locally valid surrogate objective and then improve the policy only within a region where that surrogate remains a reliable upper or lower bound on the true objective. This converts policy iteration into a constrained optimization problem that can be applied to parameterized policies such as neural networks.

The theoretical foundation is a monotonic improvement guarantee. For a previous policy $\pi_i$πi​, the method constructs a surrogate function

where $L_{\pi_i}(\pi)$Lπi​​(π) is an advantage based surrogate loss and $D_{\mathrm{KL}}^{\max}$DKLmax​ bounds the largest state dependent KL divergence between the old and new policies. The true objective is bounded by this surrogate with equality at the current policy. Therefore, if each update decreases $M_i$Mi​, the true objective improves monotonically.

For parameterized policies $\pi_\theta(a|s)$πθ​(a∣s), the authors rewrite the update in terms of parameters $\theta$θ. The theoretical penalty term involving $C D_{\mathrm{KL}}^{\max}$CDKLmax​ would produce very small and hard to tune updates. Instead, the method uses a trust region constraint:

where

This constraint controls the average KL divergence between the old and new policies over states visited by the old policy. It avoids the need to enforce a separate divergence bound at every state while still preventing large, unsafe policy changes.

The optimization problem is then estimated from samples. Starting from the surrogate loss, the objective can be written as

subject to

Here, $Q_{\theta_{\mathrm{old}}}$Qθold​​ replaces the advantage function because the two differ only by a state dependent constant that does not change the optimization direction. The action expectation is handled through importance sampling with respect to a proposal distribution $q(a|s)$q(a∣s).

The method offers two sample estimation schemes. In the single path scheme, the current policy generates one or more trajectories, and every state action pair encountered along those trajectories is used in the objective. For each visited pair, the $Q$Q value is estimated from the discounted future costs or returns along the trajectory. This scheme does not require the ability to reset the environment to arbitrary states, so it is suitable for real physical systems.

In the vine scheme, the current policy first generates longer trajectories that form trunk paths. A subset of visited states is selected as a rollout set. From each state $s_n$sn​ in this set, the method samples $K$K actions from a distribution $q$q, and for each sampled action it performs a short rollout to estimate

The variance of these estimates is reduced by using common random numbers across the $K$K rollouts, meaning that the same noise sequence is used when possible. For continuous action problems, the old policy itself is used as $q$q, while for discrete problems a uniform or broadly exploratory $q$q may be preferred. The contribution of one rollout state to the surrogate loss is estimated by the self normalized importance sampling estimator

This estimator averages over the sampled actions and then further averages over rollout states to approximate the full surrogate loss. The vine method gives lower variance advantage estimates than single path estimation, but it requires more simulator calls and requires reset access to arbitrary states. The two estimation procedures are contrasted as shown in the figure below:

The practical training loop repeatedly performs three steps. First, it collects state action pairs and Monte Carlo estimates of their values using either the single path or vine estimator. Second, it constructs an empirical version of the constrained surrogate objective and the average KL constraint. Third, it approximately solves the constrained optimization problem to update the policy parameters. The optimizer uses a Fisher information matrix formed from the analytic Hessian of the KL divergence,

rather than estimating it from outer products of sampled policy gradients. This analytic construction integrates over actions at each state and avoids storing a dense Hessian or all policy gradients for large batches. The update is obtained using a conjugate gradient step followed by a line search, where the line search ensures that the final policy satisfies the KL trust region bound.

Experiment

Experiments compare single-path and vine TRPO with ablations and baselines on MuJoCo locomotion and Atari image control, testing sampling behavior, the effect of a KL constraint, and scalability. TRPO produces high-quality locomotion gaits and outperforms prior policy-optimization methods, supporting fixed KL constraints as more robust than fixed penalties and average KL approximations as effective. On Atari, both variants learn reasonable policies from raw images, demonstrating broad applicability across large-scale, partially observed tasks.

The Atari results show that both TRPO variants learned non-trivial control from raw images and achieved reasonable scores across the seven games. They did not outperform all prior methods on every game, but they remained competitive on several tasks while using a general policy search approach rather than Atari-specific methods. Single path TRPO appears more consistently competitive overall than vine TRPO, which varies more across games. Both TRPO variants score well above random behavior across the Atari games. TRPO is competitive with prior methods on some games but weaker on others. Single path TRPO is generally more consistent than vine TRPO in these evaluations.

The Atari experiments evaluate Trust Region Policy Optimization variants trained on raw pixel observations over seven games. They show that both variants learn meaningful control policies well above random and remain competitive with prior methods on several tasks, despite not universally surpassing specialized approaches. Single-path TRPO is generally more stable and competitive across games than vine TRPO, which shows greater variability.