One-sentence Summary

Bahdanau et al. (Jacobs University Bremen and Université de Montréal) propose an attention mechanism for neural machine translation that overcomes the fixed-length vector bottleneck by letting the decoder soft-search over relevant source positions, achieving translation performance comparable to state-of-the-art phrase-based systems on English-to-French translation.

Key Contributions

• The paper introduces RNNsearch, an attention-based neural machine translation model that extends a standard encoder-decoder architecture by soft-searching over source words or encoder annotations when predicting each target word. This design removes the need to encode an entire source sentence into a single fixed-length vector and trains the alignment jointly with translation.

• RNNsearch significantly outperforms the conventional encoder-decoder model RNNencdec on English-to-French translation and is reported to be more robust to source-sentence length. The results also achieve translation quality comparable to a phrase-based statistical machine translation system.

• Qualitative analysis shows that the soft alignments produced by RNNsearch often match human intuition by linking generated target words to relevant source words or annotations. This evidence supports that the attention mechanism learns useful alignment information during translation generation.

Introduction

Neural machine translation aims to build a single large network that reads a source sentence and directly produces a translation, making machine translation more end to end than traditional phrase based systems. Early encoder decoder models compressed an entire source sentence into one fixed length vector before decoding, which limited performance on long sentences because the model had to store all information in a single bottleneck. The authors address this by introducing a model that jointly learns to align and translate, using soft search to focus on relevant source words for each generated target word, removing the fixed length bottleneck and improving translation quality, especially on longer sentences.

Dataset

• The dataset comes from the WMT ’14 English–French parallel corpora. It includes Europarl (61M words), news commentary (5.5M words), UN (421M words), and two crawled corpora (90M and 272.5M words), for a total of 850M words.

• The authors do not use monolingual data beyond these parallel corpora. To manage corpus size, they reduce the combined training data to 348M words using the data selection method of Axelrod et al. (2011), following the procedure described by Cho et al.

• The training set is the selected English–French parallel corpus. The development (validation) set is created by concatenating news-test-2012 and news-test-2013. The final evaluation is performed on the WMT ’14 test set, which contains 3003 sentences that are not present in the training data.

• For processing, the authors apply standard tokenization and build a vocabulary shortlist of the 30,000 most frequent words in each language. All out-of-vocabulary words are mapped to a special token: [UNK].

• No additional preprocessing is applied, such as lowercasing or stemming.

Method

The authors build the translation system on an encoder-decoder structure, but they replace the fixed-length summary vector used by a conventional model with a dynamic attention mechanism that is learned jointly with the language model. The overall objective remains the same: given a source sentence $\mathbf{x}=(x_1,\dots,x_{T_x})$x=(x1​,…,xTx​​), produce a target sentence $\mathbf{y}=(y_1,\dots,y_{T_y})$y=(y1​,…,yTy​​) by factorizing the sequence probability as

$$p(\mathbf{y})=\prod_{i=1}^{T_y} p(y_i\mid y_1,\dots,y_{i-1},\mathbf{x}).$$p(y)=i=1∏Ty​​p(yi​∣y1​,…,yi−1​,x).

In a standard RNN encoder-decoder, the encoder first compresses the input sentence into a single context vector, and the decoder predicts each target word using that fixed representation. The authors propose to generate, instead, a separate context vector for every target position. This allows the decoder to consult the source sequence selectively while generating the next word.

Refer to the framework diagram.

The encoder is a bidirectional recurrent neural network that reads the source sentence in both directions and produces an annotation for each source token. For position $j$j, the forward RNN yields a forward hidden state $\overrightarrow{h}_j$hj​, while the backward RNN yields a backward hidden state $\overleftarrow{h}_j$hj​. These are concatenated into an annotation vector

$$h_j=\left[\overrightarrow{h}_j^{\top};\overleftarrow{h}_j^{\top}\right]^{\top}.$$hj​=[hj⊤​;hj⊤​]⊤.

Because each annotation summarizes information from both preceding and following words, it provides a richer representation of the source context around $x_j$xj​.

The decoder maintains an RNN hidden state $s_i$si​ and predicts the next target word $y_i$yi​ according to

$$p(y_i\mid y_1,\dots,y_{i-1},\mathbf{x}) = g(y_{i-1}, s_i, c_i),$$p(yi​∣y1​,…,yi−1​,x)=g(yi−1​,si​,ci​),

where $s_i$si​ is updated by

$$s_i=f(s_{i-1},y_{i-1},c_i).$$si​=f(si−1​,yi−1​,ci​).

The key change is that $c_i$ci​ is no longer a global vector shared across all decoding steps. It is computed as a weighted combination of the source annotations:

$$c_i=\sum_{j=1}^{T_x}\alpha_{ij}h_j.$$ci​=j=1∑Tx​​αij​hj​.

The weights $\alpha_{ij}$αij​ are determined by an alignment model that scores how well the source position $j$j matches the current decoding state before emitting $y_i$yi​. The score is defined as

$$e_{ij}=a(s_{i-1},h_j),$$eij​=a(si−1​,hj​),

and the final attention weights are obtained via a softmax normalization:

$$\alpha_{ij}=\frac{\exp(e_{ij})}{\sum_{k=1}^{T_x}\exp(e_{ik})}.$$αij​=∑k=1Tx​​exp(eik​)exp(eij​)​.

In this formulation, $\alpha_{ij}$αij​ can be interpreted as the probability that the target word $y_i$yi​ is aligned to the source word $x_j$xj​, and $c_i$ci​ is the expected annotation under this distribution. Unlike traditional machine translation pipelines, alignment is not treated as a latent variable requiring discrete inference. It is a continuous, differentiable soft alignment, so gradients from the translation loss can flow directly into the alignment model and the encoder.

This design relieves the encoder of the need to encode the entire source sentence into one fixed-length vector. Instead, the information remains distributed across the sequence of annotations, and the decoder can selectively retrieve the relevant parts of the source sequence at each generation step.

Experiment

Experiments evaluate English-to-French translation on WMT ’14 data, comparing the proposed RNNsearch model with a standard RNN Encoder–Decoder under matched training and evaluation settings. The results validate the overall effectiveness of the proposed architecture, especially for long sentences, where the baseline model loses accuracy and begins omitting or altering meaning. Qualitative alignment analysis further supports the interpretation that attention weights provide useful soft alignments, capturing both monotonic correspondences and necessary non-monotonic reordering in translations.

The attention-based RNNsearch models achieve higher BLEU scores than the corresponding encoder-decoder baselines, especially on translations that contain no unknown words. Extended training of the longer RNNsearch variant further improves performance and brings it close to the phrase-based Moses baseline on known-word sentences. The gains align with the reported advantage for translating long sentences, where the baseline loses fidelity as input length grows. RNNsearch outperforms RNNencdec on both all sentences and known-word subsets. On sentences without unknown words, RNNsearch approaches the Moses system despite Moses using additional monolingual data. Training the longer RNNsearch model further improves translation quality and supports its robustness to long sentences.

The experiments compare attention-based RNNsearch models with encoder-decoder baselines and a phrase-based Moses system using BLEU scores on all sentences and subsets without unknown words. RNNsearch consistently outperforms the encoder-decoder baseline, with stronger gains on known-word translations and sentences where longer inputs reduce baseline fidelity. Further training of a longer RNNsearch variant improves performance and approaches Moses on known-word sentences, despite Moses leveraging additional monolingual data, indicating that attention improves robustness for challenging long translations.