Q-Learning is the second major method in Temporal Difference (TD) learning, and it stands out as an off-policy control algorithm. While SARSA, which we covered in the previous article, is on-policy and updates the action-value function based on the actual policy being followed, Q-learning takes a different approach by learning the optimal policy independently of the behavior policy used to generate data. The core idea behind Q-learning is to estimate the optimal action-value function, denoted as Q*(s, a), which represents the maximum expected return achievable from state s by taking action a and then following the optimal policy thereafter. This makes Q-learning particularly powerful for finding the best possible strategy, even when the agent is exploring using a suboptimal policy. The Q-learning update rule is: Q(s, a) ← Q(s, a) + α [r + γ maxₐ' Q(s', a') − Q(s, a)] This equation may look similar to SARSA’s update, but there’s a crucial difference: instead of using the actual next action a' taken in the next state s', Q-learning uses the maximum possible Q-value over all actions in s'. This greedy choice allows the algorithm to converge to the optimal policy regardless of the actions taken during exploration. In essence, Q-learning performs a form of bootstrapping, combining the Bellman optimality equation with an exponential moving average (EMA) update, much like SARSA. But while SARSA updates based on the observed next action, Q-learning looks ahead to the best possible future action, making it off-policy. This distinction is what enables Q-learning to learn the optimal policy even when the agent is acting greedily or randomly during training. It’s like learning the best strategy while experimenting with different moves—without being limited by the choices made during exploration. Q-learning is widely used in reinforcement learning due to its simplicity, convergence guarantees under certain conditions, and its ability to find optimal policies in a variety of environments. It forms the foundation for more advanced algorithms such as Deep Q-Networks (DQN) and other deep reinforcement learning methods. As we continue our journey through TD learning, keep in mind that Q-learning’s off-policy nature gives it a unique advantage: it can learn the best possible behavior even when the agent isn’t following it during training. This makes it a cornerstone of modern RL systems.