One-sentence Summary

Researchers from the Montreal Institute for Learning Algorithms (MILA) propose a lightweight framework using asynchronous gradient descent for deep reinforcement learning, presenting asynchronous variants of four RL algorithms where parallel actor-learners stabilize neural network training, with their asynchronous actor-critic method surpassing the state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU.

Key Contributions

• The paper introduces a lightweight asynchronous deep reinforcement learning framework that uses asynchronous gradient descent to optimize deep neural network controllers, including asynchronous variants of four standard reinforcement learning algorithms.

• The framework shows that parallel actor-learners updating a shared model stabilize training, enabling successful neural network controller learning across value-based and policy-based methods, off-policy and on-policy algorithms, and discrete and continuous control domains.

• The asynchronous advantage actor-critic method reaches state-of-the-art Atari performance on 16 CPU cores in half the training time of DQN on an Nvidia K40 GPU, while also succeeding on continuous motor control tasks and visual navigation in random 3D mazes.

Introduction

Deep reinforcement learning matters because it can learn control policies directly from high-dimensional observations, as in Atari games and continuous motor control tasks. However, combining online reinforcement learning with deep neural networks was often unstable because online data are correlated and non-stationary. Prior approaches used experience replay to batch and decorrelate data, but this required off-policy algorithms and added memory and computational costs. Other parallel systems improved training speed but relied on distributed setups or replay memories. The authors introduce an asynchronous reinforcement learning framework in which multiple agents act in parallel environment instances and update a shared model, producing more diverse and stationary data for stable learning. They show this enables robust on-policy and off-policy algorithms with deep networks, including asynchronous advantage actor-critic (A3C), using standard multicore CPUs.

Method

Asynchronous RL Framework

The authors leverage an asynchronous reinforcement learning framework in which multiple actor-learner threads operate in parallel on a single machine. Each thread interacts with its own copy of the environment, selects actions according to a local exploration policy, accumulates gradients over several time steps, and then applies those gradients to a shared set of network parameters. The overall design keeps the learner and actors on one machine, which removes the communication overhead of distributed parameter servers and enables Hogwild-style asynchronous updates. This makes it practical to train deep neural network policies without large hardware requirements or replay memory.

Refer to the framework described in the text: each actor-learner runs independently and periodically performs online updates to the shared model. By making each thread explore differently, the update directions applied by different actor-learners are less likely to be temporally correlated. In this way, parallel exploration provides a stabilizing role similar to that of experience replay in DQN-based training. The framework includes variants of one-step Q-learning, one-step Sarsa, n-step Q-learning, and advantage actor-critic, with each variant built around the same asynchronous actor-learner mechanism.

For one-step Q-learning and one-step Sarsa, each actor-learner observes a state, takes an action using an exploration policy, receives a reward, and computes a gradient of the temporal-difference loss. A slowly changing target network is used to compute stable targets. In the Q-learning case, the target value is formed from the discounted target-network value of the next state. In the one-step Sarsa case, the target uses the next action actually taken:

where $s'$s′ is the next state, $a'$a′ is the action selected in $s'$s′, and $\theta^{-}$θ− denotes the target network parameters. Gradients are accumulated over multiple steps before being applied. This accumulation behaves similarly to a minibatch and reduces the chance that multiple actor-learners overwrite each other’s updates. The method also allows a trade-off between computational efficiency and data efficiency.

For n-step Q-learning, the algorithm operates in a forward view. Instead of using eligibility traces or a backward view, each actor-learner first selects actions using its exploration policy for up to $t_{\max}$tmax​ steps or until a terminal state is reached. This produces a short trajectory with multiple rewards, states, and actions. The agent then computes n-step returns for each state-action pair in that trajectory. The first state receives the longest available n-step return, the next state receives a shorter n-step return, and so on, resulting in a set of updates with varying horizons. These accumulated updates are applied together in one shared-gradient step.

The advantage actor-critic variant, often called A3C, maintains two coupled components: a policy $\pi(a_t|s_t;\theta)$π(at​∣st​;θ) and a value function $V(s_t;\theta_v)$V(st​;θv​). In practice, these are implemented as two output heads of a shared neural network. The non-output layers are typically shared, while one softmax output produces action probabilities for the policy and one linear output predicts the state value. The policy and value function are updated after every $t_{\max}$tmax​ actions or when a terminal state is reached.

The policy update is based on the advantage function, which estimates how much better an action is than the expected value of its state. Using n-step returns, the advantage can be written as:

where $k$k is the effective n-step length and is upper bounded by $t_{\max}$tmax​. The corresponding policy gradient has the form:

The value function is trained toward the same n-step returns. A common critic objective is:

where $R_t$Rt​ denotes the bootstrapped n-step return used as a regression target.

The authors also incorporate an entropy regularization term in the actor-critic objective. This term encourages stochasticity in the policy and discourages premature convergence to suboptimal deterministic behavior. The full policy gradient with entropy regularization becomes:

where $H$H is the policy entropy and $\beta$β controls the strength of the entropy bonus.

The asynchronous framework supports different optimization settings. The authors compare stochastic gradient descent with momentum, RMSProp without shared statistics, and RMSProp with shared statistics across threads. The non-centered RMSProp update is:

where all operations are elementwise, $\alpha$α is a decay constant, $\eta$η is the learning rate, and $\epsilon$ϵ is a small constant for numerical stability. The variant that shares RMSProp statistics across actor-learner threads is found to be particularly robust in asynchronous training.

Experiment

The experiments evaluate the framework across Atari, TORCS, MuJoCo, and Labyrinth, showing that the asynchronous methods train efficiently on CPU cores and that A3C generally performs best, including superior Atari results, strong racing control, continuous motor tasks, and visual maze exploration. Additional analyses show that parallel actor-learners provide substantial speedups, with one-step methods also benefiting in data efficiency, and that the algorithms remain stable and robust across learning rates and initializations.

Across 57 Atari games, the asynchronous advantage actor-critic agent trained on CPU cores reaches higher mean human-normalized scores than the listed GPU-trained value methods. The feedforward version improves strongly with additional training time, moving from modest median performance to near the strongest GPU baselines while retaining the highest average score in the reported results. The four-day asynchronous feedforward agent exceeds the average performance of all listed GPU-trained DQN variants. Longer asynchronous training substantially raises both mean and median scores, especially for the feedforward agent. The prioritized DQN baseline remains strong on median score, close to the four-day asynchronous feedforward agent.

Training speedup increases consistently with more parallel actor-learners across all four asynchronous methods, with the largest gains observed at 16 threads. One-step Q-learning and one-step SARSA scale especially well, showing superlinear speedups that suggest data efficiency improvements beyond pure parallelism. All methods achieve substantial speedups as the number of parallel threads increases. One-step Q-learning and one-step SARSA scale more strongly than n-step Q and A3C. Additional parallel actor-learners help one-step methods, likely by reducing bias.

The experiments evaluate asynchronous reinforcement-learning methods on a broad Atari benchmark and through scaling studies with multiple parallel actor-learners. The Atari results show that asynchronous CPU training can achieve competitive or stronger overall performance than listed GPU-trained value methods, with longer training especially improving the feedforward agent. The scaling studies confirm that additional parallel actor-learners consistently increase training speed, with one-step variants benefiting the most and showing gains that likely reflect improvements beyond pure parallelization.