One-sentence Summary

Researchers at Universiteit van Amsterdam propose Variational Autoencoders (VAEs), introducing a reparameterization of the variational lower bound that yields a differentiable estimator optimizable via standard stochastic gradient methods, combined with an amortized recognition model fitted to the intractable posterior for efficient inference over continuous latent variables in large $\mathrm{i.i.d.}$i.i.d. datasets.

Key Contributions

• The paper introduces a reparameterization of the variational lower bound that yields the Stochastic Gradient Variational Bayes estimator, which can be differentiated and optimized with standard stochastic gradient methods for directed probabilistic models with continuous latent variables and intractable posterior distributions.

• The paper introduces Auto-Encoding Variational Bayes for i.i.d. datasets with continuous latent variables per datapoint; it fits a recognition model to the intractable posterior, amortizes inference, and scales learning to large datasets.

• The paper relates variational training of directed probabilistic models to auto-encoder learning and reports experimental evidence that the estimator and algorithm provide efficient approximate inference, while the variational objective supplies the regularization used in autoencoder variants.

Introduction

The authors address efficient approximate inference and learning in directed probabilistic models with continuous latent variables or parameters whose posteriors are intractable. This matters because many generative modeling tasks require scalable learning from large datasets, but traditional variational Bayesian methods often rely on mean-field approximations with analytically intractable expectations. Prior alternatives also have limitations: the wake-sleep algorithm requires two related but not equivalent objectives, while earlier stochastic variational methods can suffer from high variance or narrow model assumptions. The authors contribute a reparameterized, differentiable, unbiased estimator of the variational lower bound, called SGVB, and an efficient inference and learning algorithm called AEVB that trains a recognition model with standard stochastic gradient methods.

Method

Method

The method addresses a general latent variable setting. The observed data consist of independent samples, and each sample is assumed to have been generated through an unobserved continuous latent variable. Specifically, for a dataset of samples, the generative process first draws a latent variable from a prior and then generates an observed data point from a conditional likelihood. Both the prior and likelihood are assumed to belong to parametric families whose density functions are differentiable with respect to both the parameters and the latent variable.

The main difficulty is that both the true latent variables and the marginal likelihood are generally unknown. The posterior distribution of the latent variable given the data is often intractable, and the marginal likelihood may also be intractable. This situation is common when the likelihood is defined by a nonlinear model such as a neural network. The method is therefore designed to work when exact posterior inference, exact marginal likelihood evaluation, and closed-form variational inference are not available.

To overcome this, the authors introduce a recognition model, written as $q_{\phi}(\mathbf{z} \mid \mathbf{x})$qϕ​(z∣x), which approximates the true intractable posterior $p_{\boldsymbol{\theta}}(\mathbf{z} \mid \mathbf{x})$pθ​(z∣x). This recognition model is parameterized by $\phi$ϕ and is learned jointly with the parameters $\boldsymbol{\theta}$θ of the generative model. From a coding perspective, the latent variable $\mathbf{z}$z can be viewed as a compressed representation of the input. The recognition model $q_{\phi}(\mathbf{z} \mid \mathbf{x})$qϕ​(z∣x) acts as a probabilistic encoder, while the conditional distribution $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$pθ​(x∣z) acts as a probabilistic decoder.

The key quantity is the variational lower bound on the log marginal likelihood. For a single data point $\mathbf{x}^{(i)}$x(i), the log marginal likelihood can be decomposed as the sum of the KL divergence between the approximate and true posteriors and the variational lower bound:

$$\log p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)}) = D_{KL}\left(q_{\phi}(\mathbf{z} \mid \mathbf{x}^{(i)}) \,\|\, p_{\boldsymbol{\theta}}(\mathbf{z} \mid \mathbf{x}^{(i)})\right) + \mathcal{L}(\boldsymbol{\theta}, \phi; \mathbf{x}^{(i)}).$$logpθ​(x(i))=DKL​(qϕ​(z∣x(i))∥pθ​(z∣x(i)))+L(θ,ϕ;x(i)).

Since the KL divergence is nonnegative, the lower bound satisfies

$$\log p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)}) \geq \mathcal{L}(\boldsymbol{\theta}, \phi; \mathbf{x}^{(i)}).$$logpθ​(x(i))≥L(θ,ϕ;x(i)).

The lower bound can be written in two equivalent forms. The first is

$$\mathcal{L}(\boldsymbol{\theta}, \phi; \mathbf{x}^{(i)}) = \mathbb{E}_{q_{\phi}(\mathbf{z} \mid \mathbf{x}^{(i)})} \left[ -\log q_{\phi}(\mathbf{z} \mid \mathbf{x}^{(i)}) + \log p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)}, \mathbf{z}) \right].$$L(θ,ϕ;x(i))=Eqϕ​(z∣x(i))​[−logqϕ​(z∣x(i))+logpθ​(x(i),z)].

The second separates a KL regularization term from an expected log likelihood term:

$$\mathcal{L}(\boldsymbol{\theta}, \phi; \mathbf{x}^{(i)}) = - D_{KL}\left(q_{\phi}(\mathbf{z} \mid \mathbf{x}^{(i)}) \,\|\, p_{\boldsymbol{\theta}}(\mathbf{z})\right) + \mathbb{E}_{q_{\phi}(\mathbf{z} \mid \mathbf{x}^{(i)})} \left[ \log p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)} \mid \mathbf{z}) \right].$$L(θ,ϕ;x(i))=−DKL​(qϕ​(z∣x(i))∥pθ​(z))+Eqϕ​(z∣x(i))​[logpθ​(x(i)∣z)].

This decomposition is important. The KL term encourages the approximate posterior to remain close to the prior, while the expectation term measures how well the generated latent variable can explain the observed data. In model training, the goal is to maximize this lower bound with respect to both the generative parameters $\boldsymbol{\theta}$θ and the recognition parameters $\phi$ϕ.

A direct Monte Carlo estimate of the gradient with respect to $\phi$ϕ usually has very high variance. To address this, the method uses a reparameterization trick. The central idea is to express the stochastic latent variable as a deterministic function of an auxiliary noise variable. Instead of sampling $\mathbf{z}$z directly from $q_{\phi}(\mathbf{z} \mid \mathbf{x})$qϕ​(z∣x), one samples noise from a simple fixed distribution $p(\boldsymbol{\epsilon})$p(ϵ) and maps it through a differentiable transformation $g_{\phi}$gϕ​:

$$\widetilde{\mathbf{z}} = g_{\phi}(\boldsymbol{\epsilon}, \mathbf{x}), \qquad \boldsymbol{\epsilon} \sim p(\boldsymbol{\epsilon}).$$z=gϕ​(ϵ,x),ϵ∼p(ϵ).

For example, if $q_{\phi}(\mathbf{z} \mid \mathbf{x})$qϕ​(z∣x) is Gaussian with mean $\mu^{(i)}$μ(i) and diagonal variance $\sigma^2{}^{(i)}$σ2(i), then one can use

$$\mathbf{z}^{(i,l)} = \mu^{(i)} + \sigma^{(i)} \odot \boldsymbol{\epsilon}^{(l)}, \qquad \boldsymbol{\epsilon}^{(l)} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}),$$z(i,l)=μ(i)+σ(i)⊙ϵ(l),ϵ(l)∼N(0,I),

where $\odot$⊙ denotes elementwise multiplication. This shifts the randomness to the auxiliary noise variable and allows the expectation to be estimated while retaining a computational graph with respect to $\phi$ϕ.

Using this reparameterization, an unbiased stochastic estimate of an expectation under $q_{\phi}(\mathbf{z} \mid \mathbf{x}^{(i)})$qϕ​(z∣x(i)) is

$$\mathbb{E}_{q_{\phi}(\mathbf{z} \mid \mathbf{x}^{(i)})}[f(\mathbf{z})] = \mathbb{E}_{p(\boldsymbol{\epsilon})} \left[ f(g_{\phi}(\boldsymbol{\epsilon}, \mathbf{x}^{(i)})) \right] \approx \frac{1}{L} \sum_{l=1}^{L} f(g_{\phi}(\boldsymbol{\epsilon}^{(l)}, \mathbf{x}^{(i)})).$$Eqϕ​(z∣x(i))​[f(z)]=Ep(ϵ)​[f(gϕ​(ϵ,x(i)))]≈L1​l=1∑L​f(gϕ​(ϵ(l),x(i))).

This can be used to form a generic stochastic estimator of the lower bound:

$$\widetilde{\mathcal{L}}^{A}(\boldsymbol{\theta}, \phi; \mathbf{x}^{(i)}) = \frac{1}{L} \sum_{l=1}^{L} \left[ \log p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)}, \mathbf{z}^{(i,l)}) - \log q_{\phi}(\mathbf{z}^{(i,l)} \mid \mathbf{x}^{(i)}) \right],$$LA(θ,ϕ;x(i))=L1​l=1∑L​[logpθ​(x(i),z(i,l))−logqϕ​(z(i,l)∣x(i))],

where

$$\mathbf{z}^{(i,l)} = g_{\phi}(\boldsymbol{\epsilon}^{(i,l)}, \mathbf{x}^{(i)}), \qquad \boldsymbol{\epsilon}^{(l)} \sim p(\boldsymbol{\epsilon}).$$z(i,l)=gϕ​(ϵ(i,l),x(i)),ϵ(l)∼p(ϵ).

In many practical cases, the KL divergence term between the approximate posterior and the prior can be computed analytically. In that situation, only the expected log likelihood needs to be estimated by sampling. This yields a lower variance estimator:

$$\widetilde{\mathcal{L}}^{B}(\boldsymbol{\theta}, \phi; \mathbf{x}^{(i)}) = - D_{KL}\left(q_{\phi}(\mathbf{z} \mid \mathbf{x}^{(i)}) \,\|\, p_{\boldsymbol{\theta}}(\mathbf{z})\right) + \frac{1}{L} \sum_{l=1}^{L} \log p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i,l)}).$$LB(θ,ϕ;x(i))=−DKL​(qϕ​(z∣x(i))∥pθ​(z))+L1​l=1∑L​logpθ​(x(i)∣z(i,l)).

This estimator directly connects to autoencoder training. The KL term regularizes the encoder by pulling the approximate posterior toward the prior, while the second term measures reconstruction quality. The encoder produces a latent code from the input, and the decoder evaluates the probability of reconstructing the input from that code.

For a dataset with $N$N data points, the full lower bound can be estimated using a minibatch of size $M$M:

$$\mathcal{L}(\boldsymbol{\theta}, \phi; \mathbf{X}) \approx \widetilde{\mathcal{L}}^{M}(\boldsymbol{\theta}, \phi; \mathbf{X}^{M}) = \frac{N}{M} \sum_{i=1}^{M} \widetilde{\mathcal{L}}(\boldsymbol{\theta}, \phi; \mathbf{x}^{(i)}).$$L(θ,ϕ;X)≈LM(θ,ϕ;XM)=MN​i=1∑M​L(θ,ϕ;x(i)).

The scaled minibatch estimator is differentiable with respect to both $\boldsymbol{\theta}$θ and $\phi$ϕ, enabling stochastic optimization methods such as stochastic gradient descent or adaptive gradient methods. In practice, a single sample per data point can be sufficient when the minibatch is large enough.

The method becomes especially clear in the variational autoencoder case. The prior over latent variables is chosen as a centered isotropic Gaussian:

$$p_{\boldsymbol{\theta}}(\mathbf{z}) = \mathcal{N}(\mathbf{z}; \mathbf{0}, \mathbf{I}).$$pθ​(z)=N(z;0,I).

The approximate posterior is also modeled as a Gaussian with diagonal covariance:

$$q_{\phi}(\mathbf{z} \mid \mathbf{x}^{(i)}) = \mathcal{N}(\mathbf{z}; \boldsymbol{\mu}^{(i)}, \boldsymbol{\sigma}^2{}^{(i)}\mathbf{I}).$$qϕ​(z∣x(i))=N(z;μ(i),σ2(i)I).

The mean vector $\boldsymbol{\mu}^{(i)}$μ(i) and the standard deviation vector $\boldsymbol{\sigma}^{(i)}$σ(i) are produced by an encoder network from the input $\mathbf{x}^{(i)}$x(i). The latent sample is generated through the Gaussian reparameterization

$$\mathbf{z}^{(i,l)} = \mu^{(i)} + \sigma^{(i)} \odot \epsilon^{(l)}, \qquad \epsilon^{(l)} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}).$$z(i,l)=μ(i)+σ(i)⊙ϵ(l),ϵ(l)∼N(0,I).

The decoder $p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i,l)})$pθ​(x(i)∣z(i,l)) maps the latent sample back to a distribution over the observed data. Depending on the data type, the decoder may model real-valued data with a Gaussian output or binary data with a Bernoulli output.

For this Gaussian encoder and Gaussian prior, the KL term has a closed form, and the lower bound for one data point can be estimated as

$$\mathcal{L}(\boldsymbol{\theta}, \phi; \mathbf{x}^{(i)}) \approx \frac{1}{2} \sum_{j=1}^{J} \left( 1 + \log((\sigma_j^{(i)})^2) - (\mu_j^{(i)})^2 - (\sigma_j^{(i)})^2 \right) + \frac{1}{L} \sum_{l=1}^{L} \log p_{\boldsymbol{\theta}}(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i,l)}).$$L(θ,ϕ;x(i))≈21​j=1∑J​(1+log((σj(i)​)2)−(μj(i)​)2−(σj(i)​)2)+L1​l=1∑L​logpθ​(x(i)∣z(i,l)).

The first part regularizes the latent representation by keeping the encoded distribution close to the isotropic Gaussian prior. The second part encourages the decoder to reconstruct the input well from the stochastic latent code. The parameters of the encoder and decoder are trained jointly by maximizing this lower bound with stochastic gradients.

Experiment

Experiments trained encoder-decoder generative models on MNIST and Frey Face, using the variational lower bound to compare AEVB with the wake-sleep algorithm. The lower-bound results indicated that the variational objective has a regularizing effect, since superfluous latent variables did not cause overfitting. For small latent spaces, marginal likelihood estimates were used to compare optimization behavior, while low-dimensional visualizations showed that the learned encoders produce interpretable latent manifolds.