Reinforcement Learning (RL) is a powerful approach for training robots to perform complex tasks by learning from interactions with an environment. Unlike supervised learning, where the correct output is known, RL agents learn by trial and error, receiving rewards or penalties based on their actions. This makes it ideal for robotics, where the goal is to develop autonomous systems that can adapt and improve over time. In this guide, we explore three major RL paradigms—Q-Learning, Actor-Critic, and Evolutionary Algorithms—using Python and the Gym library with MuJoCo as the physics engine. We'll build a custom 3D environment for a quadruped robot (the Ant) and apply each method to teach it to jump. The default Ant-v4 environment in Gym is a 3D robot with 8 joints and 9 links, designed to move forward. To make it jump, we modify the XML file to reduce body density (making it lighter) and increase leg force. We also create a custom reward function that gives higher rewards when the robot's torso is elevated, encouraging vertical movement. To use the custom environment, we define a new class that inherits from the original AntEnv, override the step method to include our custom reward logic, and register the new environment. This allows us to treat it like any other Gym environment. Q-Learning is a foundational RL method that uses Q-values to estimate the long-term reward of taking an action in a given state. It works well in discrete action spaces. Since the Ant’s actions are continuous (force and torque on each joint), we wrap the environment in a DiscreteEnvWrapper that maps 5 discrete actions—such as moving forward, backward, or adjusting leg patterns—to continuous control signals. This allows us to use Deep Q-Networks (DQN) with Stable-Baselines3. The agent learns to jump by maximizing reward, but the motion is often jerky due to the coarse action space. Actor-Critic models, on the other hand, are better suited for continuous control. They use two components: the Actor, which selects actions, and the Critic, which evaluates the value of those actions. The Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) algorithms are among the most effective. SAC, in particular, uses two value networks to improve stability. When applied directly to the continuous CustomAntEnv-v1, the robot learns smooth, coordinated jumping behavior, far outperforming the DQN approach. For experimental alternatives, we explore Evolutionary Algorithms, such as Policy Gradients with Parameter Exploration (PGPE). Instead of training a single policy, PGPE maintains a population of policies represented as a Gaussian distribution over network weights. Each generation, it samples multiple policies, evaluates their performance, and updates the mean and variance to favor better ones. This method is highly parallel and works well in sparse reward settings. Using EvoTorch, we can evolve a policy for the Ant to jump, with the best-performing weights visualized in a rendered environment. In summary, Q-Learning is best for discrete problems and can be adapted to continuous ones with discretization, but it’s less efficient. Actor-Critic methods like SAC are the go-to for continuous control tasks, offering faster convergence and smoother behavior. Evolutionary algorithms provide a robust, parallelizable alternative, especially when reward signals are weak or delayed. The choice of method depends on the task, environment complexity, and available compute. With tools like Gym, MuJoCo, Stable-Baselines3, and EvoTorch, building and testing RL agents for robotics has never been more accessible.