One-sentence Summary

Researchers at Huazhong University of Science and Technology present an end-to-end trainable neural network that integrates feature extraction, sequence modeling, and transcription for image-based sequence recognition, achieving state-of-the-art performance on scene text benchmarks without character segmentation or predefined lexicons, and demonstrating generality through effective music score recognition.

Key Contributions

• The Convolutional Recurrent Neural Network (CRNN) unifies convolutional feature extraction, recurrent sequence modeling, and transcription into a single end-to-end trainable architecture that learns directly from word-level labels without requiring character annotations or separate preprocessing.
• The model handles sequences of arbitrary length with only height normalization, supports both lexicon-free and lexicon-based scene text recognition, and achieves superior or highly competitive performance on IIIT-5K, Street View Text, and ICDAR benchmarks.
• By eliminating fully connected layers, CRNN uses significantly fewer parameters than standard DCNN models, and its generality is confirmed by strong results on music score recognition.

Introduction

The authors tackle image-based sequence recognition, where the goal is to predict a variable-length series of labels for objects like scene text, handwriting, and musical scores. This matters because such objects are common in real-world visuals, yet standard deep convolutional neural networks (DCNNs) only handle fixed-size inputs and outputs, making them ill-suited for sequences of arbitrary length. Prior attempts either require per-character detection and cropping, which demands expensive annotations, or treat each word as a distinct class, leading to huge models that cannot generalize to other sequence types. Recurrent neural networks (RNNs) can handle sequences but rely on handcrafted feature preprocessing that breaks end-to-end training. The authors introduce the Convolutional Recurrent Neural Network (CRNN), a unified architecture that combines DCNN-based feature extraction with RNN sequence modeling, enabling direct learning from coarse sequence labels (e.g., words) without detailed character-level annotations, handling arbitrary input lengths, and training end-to-end while remaining compact and efficient.

Dataset

The authors rely on a large synthetic training set and four standard cropped-word benchmarks to train and evaluate their scene text recognition model.

• Training data

• Synth (synthetic dataset) – 8 million word images with ground-truth labels, generated by a realistic text-rendering engine (Jaderberg et al.). The model is trained once on this synthetic corpus and tested without any fine-tuning on real images.

• Test datasets (all consist of cropped word images, with lexicon-based evaluation protocols)

• ICDAR 2003 (IC03) – 251 scene images with text bounding boxes. Images with non-alphanumeric characters or fewer than three characters are ignored, yielding 860 cropped test words. Each image is paired with a per-image 50-word lexicon; a full lexicon combines all per-image sets, and an additional 50k-word lexicon is built from the Hunspell dictionary.

• ICDAR 2013 (IC13) – Inherits most data from IC03; contains 1,015 cropped word instances with ground truth.

• IIIT5k-word (IIIT5k) – 3,000 cropped word images collected from the web. Each image has a 50-word and a 1,000-word lexicon.

• Street View Text (SVT) – 249 street-view images from Google Street View, cropped into 647 word images. Each image comes with a 50-word lexicon.

• How the data is used

• The model is trained exclusively on the Synth dataset (single training phase, 8M samples).

• Evaluation is performed on the four test benchmarks without any adaptation; the network processes cropped word images and, when applicable, decodes using the provided lexicons (50-word, full, or 50k-word for IC03; 50-word, 1k-word for IIIT5k; 50-word for SVT). No additional cropping or filtering is applied beyond the preprocessing already described for the test sets.

Method

The authors propose a network architecture consisting of three main components: convolutional layers, recurrent layers, and a transcription layer. These components work together to extract features, predict label distributions, and translate them into a final sequence. The entire network can be jointly trained with a single loss function.

As shown in the figure below:

At the bottom of the network, the convolutional layers are constructed by taking convolutional and max-pooling layers from a standard CNN model, with fully-connected layers removed. Input images are scaled to the same height. The network extracts a sequence of feature vectors from the feature maps. Each feature vector is generated from left to right by concatenating the columns of all maps. Since convolution, max-pooling, and activation functions operate on local regions, they are translation invariant. Consequently, each column of the feature maps corresponds to a rectangle region of the original image, known as the receptive field.

As illustrated in the figure below:

Each vector in the feature sequence is associated with a receptive field and serves as an image descriptor for that region. This approach conveys deep features into sequential representations, making the model invariant to the length variation of sequence-like objects.

On top of the convolutional layers, a deep bidirectional Recurrent Neural Network (RNN) is built. The recurrent layers predict a label distribution $y_t$yt​ for each frame $x_t$xt​ in the feature sequence $\mathbf{x} = x_1, \dots, x_T$x=x1​,…,xT​. The authors leverage RNNs because they can capture contextual information within a sequence, back-propagate error differentials to the convolutional layers for joint training, and operate on sequences of arbitrary lengths.

To address the vanishing gradient problem of traditional RNNs, the authors employ Long-Short Term Memory (LSTM) units. An LSTM consists of a memory cell and three multiplicative gates: input, output, and forget gates.

As shown in the figure below:

Since contexts from both directions are useful in image-based sequences, the authors combine a forward and a backward LSTM into a bidirectional LSTM. Multiple bidirectional LSTMs are stacked to form a deep bidirectional LSTM, allowing for higher-level abstractions. Error differentials are propagated in the opposite direction using Back-Propagation Through Time (BPTT). A custom "Map-to-Sequence" layer bridges the convolutional and recurrent layers.

The transcription layer converts per-frame predictions into a label sequence. The authors adopt the conditional probability defined in the Connectionist Temporal Classification (CTC) layer. This probability is defined for a label sequence $\mathbf{l}$l conditioned on per-frame predictions $\mathbf{y} = y_1, \dots, y_T$y=y1​,…,yT​. It ignores the position of each label, allowing the network to be trained using only images and their corresponding label sequences without manually labeling individual character positions.

The conditional probability is formulated as: $p(\mathbf{l} | \mathbf{y}) = \sum_{\boldsymbol{\pi}: \mathcal{B}(\boldsymbol{\pi}) = \mathbf{l}} p(\boldsymbol{\pi} | \mathbf{y})$p(l∣y)=∑π:B(π)=l​p(π∣y) where $\mathcal{B}$B is a sequence-to-sequence mapping function that removes repeated labels and blanks. The probability of a path $\boldsymbol{\pi}$π is $p(\boldsymbol{\pi} | \mathbf{y}) = \prod_{t=1}^T y_{\pi_t}^t$p(π∣y)=∏t=1T​yπt​t​. This is efficiently computed using the forward-backward algorithm.

In lexicon-free mode, the sequence $\mathbf{l}^*$l∗ with the highest probability is approximated by taking the most probable label at each time stamp and mapping the result. In lexicon-based mode, the label sequence is recognized by choosing the sequence in a given lexicon $\mathcal{D}$D that has the highest conditional probability. To avoid exhaustive search over large lexicons, the authors limit the search to nearest-neighbor candidates $\mathcal{N}_\delta(\mathbf{l}')$Nδ​(l′) found using a BK-tree data structure, where $\mathbf{l}'$l′ is the sequence from lexicon-free transcription and $\delta$δ is the maximal edit distance.

The training dataset is denoted by $\mathcal{X} = \{I_i, \mathbf{l}_i\}_i$X={Ii​,li​}i​. The objective is to minimize the negative log-likelihood of the conditional probability of the ground truth: $\mathcal{O} = - \sum_{I_i, \mathbf{l}_i \in \mathcal{X}} \log p(\mathbf{l}_i | \mathbf{y}_i)$O=−∑Ii​,li​∈X​logp(li​∣yi​) where $\mathbf{y}_i$yi​ is the sequence produced by the network from $I_i$Ii​. This allows end-to-end training on pairs of images and sequences.

The network is trained with stochastic gradient descent (SGD). Gradients are calculated using back-propagation, with the forward-backward algorithm in the transcription layer and BPTT in the recurrent layers. For optimization, the authors use ADADELTA to automatically calculate per-dimension learning rates, which converges faster than the conventional momentum method.

Experiment

The evaluation first validates CRNN on scene text recognition by training solely on synthetic data and testing on four standard real-world benchmarks, then extends to optical music recognition by casting pitch recognition as a sequence prediction problem. In text recognition, CRNN consistently matches or surpasses state-of-the-art methods in constrained lexicon cases and remains promising without a lexicon, while offering end-to-end trainability, no need for character-level annotations, and a small model footprint. For musical scores, CRNN significantly outperforms commercial OMR systems, especially on noisy and real-world images, because its convolutional features resist distortions and recurrent layers capture context across notes. These findings highlight CRNN’s generality as a robust, domain-agnostic framework for image-based sequence recognition.

The CRNN network combines VGG-style convolutional layers with recurrent layers for sequence recognition. It adopts rectangular 1×2 max-pooling windows in later stages to produce wider feature maps, and inserts batch normalization after the fifth and sixth convolutional layers to accelerate training. Two stacked bidirectional LSTM layers, each with 256 hidden units, form the recurrent part of the network. Max-pooling windows of 1×2 (stride 2) are used in the third and fourth pooling layers, creating wider feature sequences and rectangular receptive fields. Batch normalization is applied after the 5th and 6th convolutional layers to speed up convergence during training. The convolutional architecture follows a VGG-VeryDeep design, feeding a map-to-sequence step before the recurrent layers.

Many existing text recognition methods are not end-to-end trainable, rely on hand-crafted features, or require character-level bounding box annotations. In contrast, CRNN integrates all compared capabilities: end-to-end training, learned convolutional features, no need for character bounding boxes, and operation without a fixed dictionary, making it a more flexible and unified approach. Several earlier methods cannot handle unconstrained text and are restricted to a predefined dictionary. CRNN is one of the few approaches that simultaneously offers end-to-end training, learned convolutional features, character-level annotation-free training, and unconstrained recognition.

The proposed CRNN model outperforms most state-of-the-art approaches in lexicon-constrained scene-text recognition, achieving top results on IIIT5k and SVT but trailing on IC03 with a full lexicon. In unconstrained settings, CRNN leads on SVT while some competitors obtain higher accuracy on IC03 and IC13, often by implicitly using a large dictionary. Many previous methods cannot operate without a lexicon at all, highlighting CRNN's broader applicability. CRNN surpasses most deep models in lexicon-driven recognition, particularly on IIIT5k and SVT, except on IC03 with the full lexicon where it falls slightly behind. For unconstrained recognition, CRNN delivers the best accuracy on SVT, whereas several methods are unable to handle the absence of a lexicon. One approach reported in the unconstrained column relies on a 90k dictionary and is not truly lexicon-free, giving it an unconstrained performance advantage.

A CRNN-based approach substantially outperforms two commercial OMR systems, Capella Scan and PhotoScore, on pitch recognition across clean, synthesized, and real-world score images. The commercial systems achieve moderate accuracy on clean data but degrade sharply under synthetic augmentations and real-world noise, whereas the CRNN maintains high fragment accuracy and very low edit distance on all datasets. Its convolutional features and recurrent context modeling make it robust to the distortions that cause binarization failures in the other systems. On clean images, the CRNN reached over 74% fragment accuracy and an edit distance of 0.37, while the commercial systems achieved only 52–55% accuracy and edit distances between 1.75 and 2.34. When tested on synthesized and real-world data, the CRNN's accuracy improved to 82–84% and its edit distance fell to 0.30, while the commercial systems' accuracy dropped to 20–44% and edit distances increased to as high as 3.05.

In scene-text recognition experiments, CRNN is compared with earlier methods on standard benchmarks and shown to unify end-to-end training, learned convolutional features, and lexicon-free operation without requiring character-level bounding boxes, achieving strong results across multiple datasets. Optical music recognition tests on clean, synthesized, and real-world score images evaluate CRNN against commercial systems Capella Scan and PhotoScore, confirming that its convolutional-recurrent design remains robust under noise and distortions that cause binarization-based competitors to degrade sharply.