One-sentence Summary

This study compares Deep Reinforcement Learning and Evolutionary Methods for continuous control by formulating parallelized versions of Proximal Policy Optimization and Deep Deterministic Policy Gradient, ultimately demonstrating through a thorough comparison of state-of-the-art techniques that neither paradigm consistently outperforms the other.

Key Contributions

• Formulate parallelized implementations of the Proximal Policy Optimization and Deep Deterministic Policy Gradient algorithms for continuous control tasks.
• Conduct a thorough comparison between state-of-the-art evolutionary strategies and deep reinforcement learning methods across continuous control domains.
• Demonstrate through experimental results that neither algorithmic family consistently outperforms the other.

Introduction

Solving continuous control problems demands robust optimization strategies, making it essential to understand how Deep Reinforcement Learning and Evolutionary Strategies stack up against each other. Previous comparative studies have largely been restricted to simple discrete environments or lack thorough evaluations of modern algorithms in both fields. To bridge this gap, the authors implement parallelized versions of state-of-the-art DRL methods like PPO and DDPG and run a comprehensive benchmark against contemporary evolutionary techniques. Their results reveal no consistent winner, highlighting that algorithm performance depends heavily on the specific control task and implementation details.

Dataset

• Dataset Composition and Sources: The authors do not construct or release a new dataset. Their experiments are built around the Pendulum control environment, with additional tasks reusing the same baseline configuration.
• Key Details for Each Subset: The study evaluates five reinforcement learning algorithms: CA3C, D3PG, P3O, NES, and CMAES. Each algorithm is paired with specific hyperparameter search ranges and empirically validated settings rather than predefined data subsets.
• Data Usage and Processing: Instead of traditional training splits or mixture ratios, the authors focus on algorithm configuration and robustness testing. CA3C, D3PG, and P3O both apply the Adam optimizer to their policy and value functions. The authors run targeted grid searches for initial learning rates between $10^{-4}$10−4 and $10^{-1}$10−1, selecting $10^{-4}$10−4 for CA3C and D3PG, and $10^{-3}$10−3 for P3O. NES requires joint tuning of variance and learning rate across ranges of $10^{-2}$10−2 to $10^{0}$100 and $10^{-3}$10−3 to $10^{0}$100, with $0.1$0.1 for both parameters delivering optimal performance. CMAES uses a standard deviation search from $10^{-2}$10−2 to $10^{1}$101, where a value of $1$1 yields the best results.
• Additional Processing Details: The provided text does not mention data cropping, metadata construction, or subset filtering. Due to computational constraints, the authors only perform thorough grid searches for one or two critical hyperparameters per algorithm on the Pendulum task. All remaining parameters are either adjusted empirically or left at their package defaults, which also serves to demonstrate the algorithms robustness to hyperparameter variations.

Method

The authors leverage a unified framework for evaluating various reinforcement learning and evolutionary algorithms, where the core objective is to optimize a policy $\pi$π parameterized by $\theta$θ to maximize the expected discounted return in a Markov decision process. The policy $\pi$π can be either stochastic or deterministic, and in most cases, $\theta$θ corresponds to the weights of a neural network. For deep reinforcement learning methods such as Actor-Critic (CA3C) and Parallelized Proximal Policy Optimization (P3O), the network architecture is predefined, and $\theta$θ represents only the weights. In contrast, for neuroevolutionary approaches like Covariance Matrix Adaptation Evolution Strategy (CMAES) and Natural Evolution Strategy (NES), $\theta$θ also refers to the network weights under a fixed topology. However, in NeuroEvolution of Augmenting Topologies (NEAT), $\theta$θ encompasses both the network structure and weights, enabling the evolution of both connectivity and parameter values.

In the deep reinforcement learning setting, the policy is typically parameterized as a multivariate Gaussian distribution in continuous action spaces, with mean $\boldsymbol{\mu}(s, \theta)$μ(s,θ) and covariance $\boldsymbol{\Sigma}(s, \theta)$Σ(s,θ). For stability and simplicity, the covariance is often set to the identity matrix, $\boldsymbol{\Sigma}(s, \theta) \equiv \boldsymbol{I}$Σ(s,θ)≡I, effectively removing entropy regularization. The value function $v_{\pi}(s)$vπ​(s) and action-value function $q_{\pi}(s, \boldsymbol{a})$qπ​(s,a) are approximated using neural networks, and the policy gradient theorem is applied to compute the gradient of the objective $J(\theta) = v_{\pi}(s_0)$J(θ)=vπ​(s0​), which is then used to update the policy parameters.

Actor-Critic (CA3C) employs a dual-network architecture with an actor and a critic. The actor outputs the policy distribution, while the critic estimates the state value function. The policy update follows the policy gradient theorem, using the advantage function derived from the difference between the immediate reward and the value function estimates. In continuous control, the actor policy is often modeled as a Gaussian distribution, and the critic is trained via semi-gradient temporal difference learning. To enhance data efficiency and reduce variance, the method incorporates asynchronous parallelization, where multiple agents interact with the environment independently and contribute to gradient computation in a non-centered manner.

Parallelized Proximal Policy Optimization (P3O) extends the Proximal Policy Optimization (PPO) framework by incorporating parallelization. P3O uses the clipped objective function $L^{CLIP}(\theta) = \mathbb{E}[\min(r_t(\theta)A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)A_t)]$LCLIP(θ)=E[min(rt​(θ)At​,clip(rt​(θ),1−ϵ,1+ϵ)At​)], which constrains policy updates to prevent large deviations from the previous policy. The advantage function is computed using a truncated generalized advantage estimate, combining temporal difference errors over a trajectory with a discount factor $\gamma$γ and a lambda parameter $\lambda$λ. P3O maintains individual experience replay buffers for each worker and performs a single batch update per iteration to enhance stability. Unlike Distributed PPO (DPPO), which may drop gradients during synchronization, P3O uses a simpler synchronized gradient update with a shared lock, ensuring all gradients are applied. The value function is updated using semi-gradient TD learning.

Distributed Deep Deterministic Policy Gradient (D3PG) adapts the DDPG algorithm for distributed training. The deterministic policy gradient theorem is applied to compute gradients of the policy parameters, with the actor network outputting actions and the critic network estimating the action-value function. To stabilize learning, D3PG employs experience replay and target networks. In the distributed setting, multiple workers interact with the environment and contribute transitions to a shared replay buffer. At each update step, workers sample batches from this buffer, compute gradients using the DDPG algorithm, and synchronize updates. The target networks are also shared across workers, ensuring consistency in the learning process.

CMAES operates as a population-based optimization method, where each generation consists of candidate parameter vectors $\theta_i$θi​ sampled from a multivariate Gaussian distribution $\theta_i \sim \mu + \sigma \mathcal{N}(\mathbf{0}, \Sigma)$θi​∼μ+σN(0,Σ). The mean $\mu$μ, step size $\sigma$σ, and covariance matrix $\Sigma$Σ are updated based on the performance of evaluated candidates, guiding the search toward better solutions. NES, on the other hand, models the population distribution $p_{\phi}(\theta)$pϕ​(θ) as a Gaussian with mean $\phi$ϕ and fixed covariance $\sigma^2 I$σ2I. The update rule is derived from the gradient of the expected fitness, leading to a policy update that incorporates noise perturbations scaled by the fitness values. NEAT evolves both network structure and weights through genetic operations, using innovation numbers and historical markings to manage topology evolution efficiently.

Experiment

This study systematically compares state-of-the-art deep reinforcement learning and evolutionary strategy algorithms across diverse continuous control tasks to validate their learning efficiency, stability, and architectural scalability. The experimental results demonstrate that performance is highly task-dependent, with evolutionary methods excelling in careful exploration and exhibiting greater stability, while deep reinforcement learning approaches better manage complex dynamics and scale more effectively with larger networks. Ultimately, the findings highlight the complementary nature of both paradigms, indicating that optimal algorithm selection should be guided by specific task requirements rather than seeking a universally dominant solution.

The authors compare deep reinforcement learning and evolutionary strategies in continuous control tasks, evaluating their performance across different environments. Results show that deep RL methods generally outperform evolutionary methods in terms of data efficiency and learning speed, but evolutionary methods demonstrate better stability and exploration in tasks requiring careful exploration. The performance of both approaches varies significantly depending on the task complexity and network size. Deep RL methods are more data-efficient and faster in learning compared to evolutionary methods in most tasks. Evolutionary methods exhibit greater stability and are better at exploration, particularly in tasks requiring careful navigation. The performance of deep RL methods improves with larger network sizes, while evolutionary methods show inconsistent scaling with network complexity.

The authors compare deep reinforcement learning and evolutionary strategies in continuous control tasks using a set of benchmark environments. Results show that deep RL methods generally outperform evolutionary methods in data efficiency and learning speed for simpler tasks, while evolutionary methods exhibit better stability and exploration in tasks requiring careful navigation. The performance of both approaches varies significantly depending on the task complexity, with deep RL methods handling rich dynamics better and evolutionary methods showing superior performance in exploration-heavy scenarios. Deep RL methods achieve faster learning and better data efficiency in simpler tasks compared to evolutionary methods. Evolutionary methods demonstrate superior stability and exploration, particularly in tasks requiring careful navigation. Deep RL methods scale better with larger network sizes, while evolutionary methods show inconsistent performance with increased complexity.

The study compares deep reinforcement learning and evolutionary strategies across continuous control benchmark environments to validate their performance under varying task complexities and network scales. The experiments demonstrate that deep reinforcement learning offers superior data efficiency and faster learning, particularly in simpler tasks and when paired with larger networks. Conversely, evolutionary strategies provide greater stability and more effective exploration, making them better suited for complex scenarios requiring careful navigation. Overall, the findings indicate that method selection should align with specific task demands, as each approach excels in different computational and environmental contexts.