One-sentence Summary

Stanford University proposes global and local attention mechanisms for neural machine translation, demonstrating that local attention yields a 5.0 BLEU improvement over non-attentional systems and that an ensemble of different attention architectures achieves a new state-of-the-art 25.9 BLEU on WMT'15 English-to-German translation.

Key Contributions

• This paper introduces two attentional mechanisms for neural machine translation: a global mechanism that attends to all source positions and a local mechanism that attends to only a subset of source positions. The local mechanism is differentiable and computationally less expensive than global or soft attention.
• This paper compares various alignment functions and attention configurations, identifying which functions are best for global and local attention and analyzing effects on learning, handling of long sentences, alignment quality, and translation outputs.
• On WMT English-German translation tasks, local attention yields gains of up to 5.0 BLEU over non-attentional systems that incorporate dropout, and an ensemble using different attention architectures reaches 25.9 BLEU on WMT'15 English-German, surpassing the prior best NMT and n-gram reranker system by 1.0 BLEU.

Introduction

Neural machine translation became a leading approach for large scale translation because it trains end to end, requires minimal domain knowledge, and avoids the large phrase tables and complex decoders used by traditional systems. Early NMT models still faced challenges in capturing word alignments and handling long sentences, especially when the model had to emit target words without explicitly deciding which source words mattered most. The authors leverage attention mechanisms, building on prior work that learns alignments between source and target words, to design two simpler attention based NMT models: a global model that considers all source words and a local model that attends to a subset of source positions. They also study different alignment functions for these models. Their experiments show strong gains on English and German WMT tasks, including up to 5.0 BLEU over non attentional systems and new state of the art results for English to German translation.

Method

The proposed system uses an encoder-decoder neural machine translation pipeline. The encoder reads a source sequence $x_1,\ldots,x_n$x1​,…,xn​ and produces source hidden states, while the decoder generates a target sequence $y_1,\ldots,y_m$y1​,…,ym​ one word at a time. The authors leverage a stacking recurrent architecture, using LSTM cells for both the encoder and the decoder. The conditional generation probability factorizes as:

$$\log p(y \mid x) = \sum_{j=1}^{m} \log p(y_j \mid y_{<j}, \boldsymbol{s})$$logp(y∣x)=j=1∑m​logp(yj​∣y<j​,s)

where $\boldsymbol{s}$s denotes the information carried from the source side into the decoder. In the base recurrent design, each decoder hidden state is updated according to:

$$\boldsymbol{h}_j = f(\boldsymbol{h}_{j-1}, \boldsymbol{s}), \qquad p(y_j \mid y_{<j}, \boldsymbol{s}) = \operatorname{softmax}(g(\boldsymbol{h}_j))$$hj​=f(hj−1​,s),p(yj​∣y<j​,s)=softmax(g(hj​))

Here, $f$f is the recurrent transition function and $g$g projects the decoder state to the target vocabulary. Training uses the standard parallel sequence objective:

$$\mathcal{J}_t = \sum_{(x,y)\in D} -\log p(y \mid x)$$Jt​=(x,y)∈D∑​−logp(y∣x)

The framework diagram below shows the recurrent encoder stack reading the source sequence and the recurrent decoder stack producing target words step by step.

Attention modules modify this pipeline by allowing the decoder to consult source hidden states throughout decoding, rather than using only a fixed summary at initialization. Across the attention variants, the computation at each decoding step follows a common pattern. Given the top decoder hidden state $\boldsymbol{h}_t$ht​ and a source side context vector $\boldsymbol{c}_t$ct​, the model forms an attentional hidden state:

$$\tilde{\boldsymbol{h}}_t = \tanh\left(\boldsymbol{W}_c[\boldsymbol{c}_t;\boldsymbol{h}_t]\right)$$h~t​=tanh(Wc​[ct​;ht​])

and then predicts the next target word:

$$p(y_t \mid y_{<t}, x) = \operatorname{softmax}(\boldsymbol{W}_s \tilde{\boldsymbol{h}}_t)$$p(yt​∣y<t​,x)=softmax(Ws​h~t​)

The difference between the attention designs lies in how $\boldsymbol{c}_t$ct​ is computed from the encoder hidden states.

In the global attention model, the decoder attends to every source hidden state $\bar{\boldsymbol{h}}_s$hˉs​. For each target step $t$t, the model computes a variable length alignment vector $\boldsymbol{a}_t$at​, where:

$$a_t(s) = \frac{\exp\left(\operatorname{score}(\boldsymbol{h}_t,\bar{\boldsymbol{h}}_s)\right)} {\sum_{s'} \exp\left(\operatorname{score}(\boldsymbol{h}_t,\bar{\boldsymbol{h}}_{s'})\right)}$$at​(s)=∑s′​exp(score(ht​,hˉs′​))exp(score(ht​,hˉs​))​

The alignment function is then used to form the context vector as a weighted average over all source states. The authors consider several score functions:

$$\operatorname{score}(\boldsymbol{h}_t,\bar{\boldsymbol{h}}_s)= \begin{cases} \boldsymbol{h}_t^{\top}\bar{\boldsymbol{h}}_s, & \text{dot product}\\ \boldsymbol{h}_t^{\top}\boldsymbol{W}_a\bar{\boldsymbol{h}}_s, & \text{general}\\ \boldsymbol{W}_a[\boldsymbol{h}_t;\bar{\boldsymbol{h}}_s], & \text{concatenation} \end{cases}$$score(ht​,hˉs​)=⎩⎨⎧​ht⊤​hˉs​,ht⊤​Wa​hˉs​,Wa​[ht​;hˉs​],​dot productgeneralconcatenation​

This global design follows a compact path from the current decoder state to alignment weights, then to the context vector, and finally to the attentional output state. The attention mechanism diagram below shows how the global weights are produced and combined into the context vector.

Global attention considers the full source sequence at each decoding step, which becomes costly for longer inputs. To reduce this expense, the authors design a local attention module that focuses on a small window around a predicted alignment center $p_t$pt​. For each target word, the context vector is computed only from source hidden states in the interval:

$$[p_t-D,\; p_t+D]$$[pt​−D,pt​+D]

The local alignment vector therefore has fixed dimensionality $2D+1$2D+1. Two variants are used. The monotonic variant sets $p_t = t$pt​=t, assuming approximate monotonicity between source and target order. The predictive variant instead estimates the alignment center from the decoder state:

$$p_t = S \cdot \operatorname{sigmoid}(\boldsymbol{v}_p^{\top}\tanh(\boldsymbol{W}_p\boldsymbol{h}_t))$$pt​=S⋅sigmoid(vp⊤​tanh(Wp​ht​))

where $S$S is the source sentence length, and $\boldsymbol{W}_p$Wp​ and $\boldsymbol{v}_p$vp​ are learned parameters. To favor source positions near the predicted center, the local weights include a Gaussian prior:

$$\boldsymbol{a}_t(s)= \operatorname{align}(\boldsymbol{h}_t,\bar{\boldsymbol{h}}_s) \exp\left(-\frac{(s-p_t)^2}{2\sigma^2}\right), \qquad \sigma=\frac{D}{2}$$at​(s)=align(ht​,hˉs​)exp(−2σ2(s−pt​)2​),σ=2D​

This mechanism keeps the model differentiable while limiting attention to a local region. The local attention diagram below shows the predicted aligned position and the windowed source states used to form the context vector.

The global and local attention variants initially compute attention at each step without explicitly feeding past alignment information forward. To make the decoder aware of earlier alignment decisions, the authors introduce an input feeding connection. The attentional vector $\tilde{\boldsymbol{h}}_t$h~t​ is concatenated with the input representation at the next step:

$$\boldsymbol{z}_{t+1} = [\boldsymbol{e}_{t+1}; \tilde{\boldsymbol{h}}_t]$$zt+1​=[et+1​;h~t​]

where $\boldsymbol{e}_{t+1}$et+1​ is the next input embedding or input state. This connection acts as a coverage mechanism: the decoder can reuse information about previously attended source positions, and the recurrent network also gains additional depth across time and layer boundaries. The approach can be applied to general stacking recurrent architectures, including architectures without attention. The input feeding diagram below illustrates how attentional outputs from one step become part of the inputs for later steps.

Experiment

The experiments evaluate stacking LSTM neural machine translation systems on WMT English-German translation in both directions, using WMT training data, development sets for hyperparameter selection, and standard test sets scored with tokenized and NIST BLEU. Systematic ablations on English-German and German-English tasks show that source reversal, dropout, attention, input feeding, improved alignment functions, and unknown-word replacement contribute progressively, while model ensembles achieve strong performance on the newer test set. Additional analyses examine learning curves, sentence length effects, attention architecture choices, alignment quality, and sample outputs, indicating that attention models generalize better to longer inputs, handle rare and named entities more accurately, and learn useful alignments, although alignment quality and translation quality are not perfectly correlated.

The English-German results show that reversing the source sentence and adding dropout improve the non-attentional baseline, after which attention mechanisms provide the largest gains. Global attention makes the model competitive with a strong attentional system, while input feeding and local attention with predictive alignments yield further improvements and unknown word replacement adds a useful gain. An ensemble of several model variants reaches the best reported BLEU score, indicating that combining attention choices improves translation quality. Progressive model changes produce steady gains in BLEU and lower perplexity. Local attention with predictive alignments outperforms global attention on this task. Unknown word replacement benefits most when attention provides useful source-target links. Ensembling diverse attention and regularization settings achieves the top result.

On newstest2015, the best ensemble model with unknown-word replacement achieves a new state-of-the-art English-German translation score, exceeding the previous best NMT system with a 5-gram reranker by 1.0 BLEU. This result shows that the model generalizes well to a newer test set even when trained on slightly less data than in the earlier setting. The 8-model ensemble with unknown-word replacement reaches 25.9 BLEU. It outperforms the prior NMT plus 5-gram reranker system by 1.0 BLEU. The model transfers well from its training setting to newstest2015.

The German-English results show that attention improves translation quality over a reversed-source base system, and input feeding gives an additional improvement. More effective alignment choices and unknown word replacement provide further gains, with attention especially helping rare words. Global attention improves BLEU over the reversed-source base system. Input feeding gives another relative gain on top of global attention. Unknown word replacement further increases translation quality by improving rare word alignment.

The results show that attention scope and alignment function matter: dot works better with global attention, while general is stronger for local attention. Location-based alignment gives only modest gains from unknown word replacement, and one local-m dot setup is clearly less successful. Predictive local attention is the strongest overall configuration in both perplexity and translation quality. Predictive local attention performs best overall. Dot alignment benefits global attention, while general alignment favors local attention. Location-based alignment yields only small improvements from unknown word replacement.

The table compares alignment error rates for attention-based translation models and the Berkeley aligner on RWTH English-German data. Local attention models produce better alignment quality than the global model, while the ensemble is strong but not superior to the best local model. The attentional models reach scores close to the Berkeley aligner even though the analysis extracts only one-to-one alignments. Local attention gives lower AER than global attention Ensemble aligns well but does not beat local-m on AER Attentional models approach Berkeley aligner quality using one-to-one alignments

Experiments evaluate a range of non-attentional and attentional neural machine translation configurations on English-German and German-English tasks. Progressive ablations show that reversed-source baselines benefit from dropout, while attention, especially local attention with predictive alignments, provides the largest translation gains, with input feeding and unknown-word replacement offering further improvements. Additional analyses indicate that alignment quality is closely tied to attention scope and that ensembles of diverse attention and regularization settings yield the strongest overall performance.