One-sentence Summary

Researchers at the University of Amsterdam, OpenAI, and the University of Toronto propose Adam, a first-order stochastic optimization method based on adaptive estimates of lower-order moments that is computationally efficient, memory-light, requires minimal hyperparameter tuning, and performs well on large-scale problems with noisy or sparse gradients.

Key Contributions

• The paper introduces Adam, a computationally efficient first-order optimizer for stochastic objective functions that uses adaptive estimates of first and second gradient moments and is suited to large datasets, high-dimensional parameters, non-stationary objectives, noisy gradients, and sparse gradients.
• The paper analyzes convergence and derives a regret bound under the online convex optimization framework, and explains Adam's relation to RMSProp and AdaGrad, including how bias correction prevents excessively large parameter updates when beta_2 is near 1.
• Experimental results show Adam compares favorably with other stochastic optimization methods on non-convex machine-learning optimization problems, and the paper introduces AdaMax, a variant based on the infinity norm.

Introduction

Stochastic first-order optimization is central to machine learning and scientific computing because many objectives are defined over large, noisy datasets or regularizers such as dropout. In high-dimensional parameter spaces, second-order methods are often impractical, while earlier first-order optimizers have tradeoffs: AdaGrad handles sparse gradients well, and RMSProp suits online or non-stationary settings, but RMSProp without bias correction can produce unstable updates when sparse-gradient behavior is needed. The authors introduce Adam, which uses running estimates of the first and second moments of gradients, with bias correction, to compute per-parameter adaptive learning rates. Adam combines sparse-gradient robustness with non-stationary adaptivity, requires little memory, and provides convergence analysis and strong empirical results on large-scale machine learning problems.

Method

The authors design the method as an adaptive stochastic optimization algorithm for minimizing an expected objective function. Let $f(\theta)$f(θ) be a noisy scalar objective that is differentiable with respect to parameters $\theta$θ, and suppose the algorithm observes a sequence of stochastic realizations $f_1(\theta), \ldots, f_T(\theta)$f1​(θ),…,fT​(θ), for example through minibatches or through inherently noisy gradient estimates. At timestep $t$t, the gradient is

The central idea is to maintain running estimates of both the first and second moments of the gradient. Instead of using the raw gradient alone, the optimizer normalizes a smoothed gradient direction by an estimate of gradient variance. This produces parameter updates that are adaptive in scale and robust to noisy or sparse gradients.

The core update can be written as a repeated module. The optimizer begins with $\theta_0$θ0​, $m_0 = 0$m0​=0, and $v_0 = 0$v0​=0, where $m_t$mt​ tracks the gradient and $v_t$vt​ tracks the squared gradient. At each step,

Here, $\beta_1$β1​ and $\beta_2$β2​ control the exponential decay rates of the first and second moment estimates. The hyperparameter $\alpha$α controls the base stepsize, and $\widehat{\epsilon}$ϵ provides numerical stability in the denominator.

The term $m_t$mt​ is an exponential moving average of the gradient and serves as an estimate of the first moment. The term $v_t$vt​ is an exponential moving average of the elementwise squared gradient and serves as an estimate of the second raw moment. Because both averages are initialized at zero, their early values are biased toward zero. This bias is especially pronounced when $\beta_1$β1​ or $\beta_2$β2​ is close to $1$1, since old values are heavily discounted and the initial zeros persist for many steps.

To counteract this, the method divides the moving averages by the terms $1-\beta_1^t$1−β1t​ and $1-\beta_2^t$1−β2t​. The bias correction for the second moment can be understood by writing the recursive average as

Taking expectations gives

with additional discrepancy that can remain small when the decay rate gives little weight to very old gradients. Dividing by $1-\beta_2^t$1−β2t​ removes the initialization bias and yields a scale correct estimate $\widehat{v}_t$vt​. The same logic applies to the first moment.

The resulting update uses the bias corrected first moment and the square root of the bias corrected second moment. The effective parameter change can be viewed as

The ratio

acts like a signal to noise ratio. When the smoothed gradient estimate is consistent across recent steps, the magnitude of this ratio can be close to one, allowing larger effective moves. When the gradient direction is noisy or uncertain, the ratio shrinks, reducing the effective step. Near an optimum, the signal to noise ratio often becomes small, which naturally reduces the update magnitude. This behavior gives the optimizer a form of automatic annealing without requiring an external stepsize schedule in many practical settings.

The update also has a bounded effect. For the common case where $(1-\beta_1) \leq \sqrt{1-\beta_2}$(1−β1​)≤1−β2​​, the effective step is bounded by approximately $\alpha$α. In more severe sparse cases, a looser bound is

This bounded step behavior creates a trust region around the current parameters: the optimizer does not allow a single noisy gradient estimate to produce arbitrarily large movements. Because $\alpha$α controls the upper scale of parameter changes, it is often possible to choose its magnitude based on prior knowledge of the parameter geometry, such as expected distance from initialization to good solutions.

Another important property is scale invariance. If gradients are rescaled by a constant factor $c$c, then $m_t$mt​ scales by $c$c and $v_t$vt​ scales by $c^2$c2. The ratio

remains unchanged. Therefore, multiplying all gradients by a constant does not change the optimizer direction or step magnitude, except for the numerical stabilizer.

The computational flow can be simplified by folding the bias correction terms into the stepsize. Instead of computing $\widehat{m}_t$mt​ and $\widehat{v}_t$vt​ separately, one can use

and then update

This formulation is mathematically equivalent to the bias corrected update while avoiding an extra division in the main loop.

The method is also analyzed through an online learning framework. Given an arbitrary sequence of convex cost functions $f_1(\theta), \ldots, f_T(\theta)$f1​(θ),…,fT​(θ), the algorithm chooses parameters $\theta_t$θt​ before observing the current cost. The performance is measured by regret relative to the best fixed parameter $\theta^*$θ∗:

Under bounded gradient and bounded parameter movement assumptions, the update rule achieves an $O(\sqrt{T})$O(T​) regret bound. Consequently, the average regret satisfies

which implies that the average regret converges to zero as $T$T grows. For sparse data, the adaptive normalization can improve over non adaptive methods because dimensions that are rarely updated should not receive the same scaling treatment as frequently updated dimensions.

An important variant arises by generalizing the second moment normalization from an $L^2$L2 based quantity to an $L^p$Lp based quantity. If the stepsize is made inversely proportional to an $L^p$Lp norm surrogate, the second moment estimator can be written as

Taking the limit as $p$p approaches infinity yields a simpler recursive quantity:

This quantity behaves like a decayed running maximum of gradient magnitudes. Using this in place of $\sqrt{\widehat{v}_t}$vt​​ leads to a more stable variant for settings where large norm generalizations become numerically difficult. In this formulation, initialization bias correction is not required, and the effective parameter update has the simpler bound

The core training module can also be augmented with temporal averaging. Since stochastic optimization produces noisy iterates, averaging recent parameters often improves generalization. A simple extension is to maintain an exponential moving average of the parameter vector:

with $\bar{\theta}_0 = 0$θˉ0​=0. Bias correction can again be applied using

Alternatively, a uniform average over iterates can be used, but the exponential form places more weight on more recent parameters and fits naturally into the streaming gradient optimization loop.

Experiment

The experiments evaluate Adam across logistic regression, multilayer neural networks, convolutional neural networks, and a variational autoencoder, comparing it with momentum SGD, Adagrad, SFO, and RMSProp under consistently tuned settings. Logistic regression experiments confirm that Adam converges comparably to momentum SGD and handles sparse features effectively, matching Adagrad. On non-convex deep models, Adam generally trains efficiently, often outperforming or matching competing optimizers, while in convolutional networks it benefits from adaptive layer-wise scaling and variance reduction through momentum. The bias-correction study shows that this term improves stability when slow-decay estimates are used, making Adam at least as strong as RMSProp across the tested configurations.