One-sentence Summary

Fully Convolutional Networks (FCN), introduced by UC Berkeley, adapt contemporary classification architectures into end-to-end pixel-to-pixel segmentation models that combine deep coarse semantic information with shallow fine appearance information via a skip architecture, achieving state-of-the-art results on PASCAL VOC, NYUDv2, and SIFT Flow with inference under one fifth of a second per image on arbitrary input sizes.

Key Contributions

• Fully convolutional networks are formulated for end-to-end semantic segmentation from whole-image inputs to dense pixel outputs, with arbitrary input sizes producing correspondingly sized predictions, and pretrained AlexNet, VGG, and GoogLeNet models are adapted through fully convolutional fine-tuning.
• A skip architecture is defined to combine deep, coarse semantic information with shallow, fine appearance information, producing a locally precise and globally consistent segmentation representation trained end-to-end.
• The resulting models achieve state-of-the-art segmentation on PASCAL VOC, NYUDv2, and SIFT Flow, including a 20% relative improvement to 62.2% mean IU on PASCAL VOC 2012, and complete inference in less than one fifth of a second for a typical image.

Introduction

Convolutional networks had already advanced visual recognition from whole-image classification toward more local structured tasks, making pixel-level semantic segmentation a natural next step for detailed scene understanding. Prior approaches often relied on patchwise training, small models without supervised pretraining, or additional pre and post processing such as superpixels, proposals, random fields, local classifiers, input shifting, multi-scale processing, and ensembles. These choices made learning and inference less efficient and prevented many models from operating as simple end-to-end systems.

The authors leverage the insight that classification convolutional networks can be reinterpreted as fully convolutional networks and fine-tuned for dense pixelwise prediction. They train networks end-to-end on whole images, using in-network upsampling and a skip architecture that combines deep semantic features with shallow fine-grained appearance and location cues. This approach enables efficient arbitrary-size input processing and achieves state-of-the-art semantic segmentation without the extra machinery used in earlier systems.

Method

The authors leverage a fully convolutional reinterpretation of standard recognition networks to turn classification models into dense predictors. In a conventional convolutional network, each layer is a three-dimensional array of size $h \times w \times d$h×w×d, where $h$h and $w$w are spatial dimensions and $d$d is the channel dimension. Convolution, pooling, and activation operations are local and translation invariant, so the output at position $(i,j)$(i,j) in a layer can be written as

where $k$k is the kernel size, $s$s is the stride, and $f_{ks}$fks​ denotes the layer type. Composition of such layers preserves this local, stride-aware structure, yielding a deep filter rather than a fixed-size classifier. The composition rule is

This property allows the network to process inputs of arbitrary spatial size while producing outputs whose spatial dimensions are determined by the network stride. Refer to the transformation diagram.

The method begins by convolutionalizing an existing classification backbone. The fully connected layers are reinterpreted as convolutions whose kernels span the entire input region. Once the terminal classifier is removed, the network no longer produces a single image-level label; instead, it produces a spatial map of class scores. The authors then append a $1 \times 1$1×1 convolution to predict class scores at each coarse output location, followed by an upsampling path that connects coarse predictions back to the pixel grid. The original classification weights are retained and fine-tuned end-to-end for dense prediction.

Because segmentation requires supervision at every output cell, the loss is formulated as a sum over the spatial locations of the final layer:

Under this spatial loss, gradient computation over the full image is equivalent to optimizing over the set of final-layer receptive fields. When those receptive fields overlap substantially, processing the whole image layer by layer is much more efficient than computing each patch independently. The authors therefore train whole images end to end with a pixelwise loss, while ignoring ground-truth pixels marked as ambiguous or masked. They note that patchwise training can be viewed as loss sampling, but whole-image fully convolutional training is both efficient and effective.

A key module in the architecture is learned upsampling. Coarse outputs are produced at reduced spatial resolution because classification networks typically include pooling and strided convolution to keep computation manageable. The authors connect these coarse predictions to dense pixels using deconvolution, which can be interpreted as backwards strided convolution. An upsampling factor $f$f corresponds to a convolution with fractional input stride $1/f$1/f. Rather than fixing the interpolation rule, the deconvolution filter is allowed to be learned by backpropagation, with activation functions enabling nonlinear upsampling if desired. In practice, the upsampling layers may be initialized to bilinear interpolation and then refined during training.

Although the baseline fully convolutional classifier can be adapted for segmentation, its final prediction is coarse. In the common stride-32 version, often called FCN-32s, the output prediction has a 32-pixel receptive field stride, limiting the spatial detail that can be recovered after upsampling. To refine the output, the authors combine predictions from multiple depths in the feature hierarchy. The resulting architecture is a directed acyclic graph rather than a strictly linear stack. It preserves a single main stream that extracts semantic information, while adding skip paths from earlier layers that retain finer spatial detail. As shown in the figure below.

The first refinement creates FCN-16s. The network already produces a prediction at stride 32 from the deepest convolutional feature layer. The authors add a $1 \times 1$1×1 convolution on top of the $pool4$pool4 layer, which has stride 16, to produce an additional class prediction. This stride-16 prediction is fused with the stride-32 prediction by upsampling the latter by a factor of 2 and summing the two score maps. The network then performs another upsampling step to reach pixel resolution. The new skip-layer parameters are initialized so that the fused prediction begins close to the original FCN-32s behavior, while the upsampling parameters may be initialized to bilinear interpolation. The added weights are learned at a reduced learning rate.

The next refinement creates FCN-8s. The network adds another prediction at stride 8 from the $pool3$pool3 layer. This stride-8 prediction is fused with the already-combined stride-16 prediction after an additional $2\times$2× upsampling step. The combined score map is then upsampled to full pixel resolution. This hierarchical fusion lets the model use coarse, high-level semantics to determine what should be present while using shallower, high-resolution features to determine where object boundaries and local structures lie. The authors call this nonlinear feature hierarchy the deep jet because it combines scale information across layers.

The effect of this multi-scale fusion is increased segmentation precision without abandoning the semantic robustness of the deep backbone. FCN-32s provides globally coherent but coarse predictions, FCN-16s recovers finer structure, and FCN-8s further improves local boundary quality. The figure below contrasts the progressively refined output with the ground truth.

The overall method therefore transfers classification-oriented representations to dense prediction through four linked ideas: convolutionalization of classifiers, whole-image training with a pixelwise loss, learned in-network upsampling, and skip fusion of hierarchical feature maps. The resulting design supports end-to-end learning from arbitrary-sized inputs to pixel-level outputs while preserving the computational efficiency of convolutional feature extraction.

Experiment

The experiments evaluate fully convolutional segmentation models trained by fine-tuning ILSVRC classifiers and augmenting them with in-network upsampling and skip connections, using PASCAL VOC for segmentation and extending to NYUDv2 scene parsing and SIFT Flow joint semantic and geometric prediction. Ablations in the experimental framework show that whole-image training is sufficient, random patch sampling does not substantially improve convergence, mild label imbalance does not require class balancing, simple augmentation and four-channel early fusion provide limited benefit, while expanded annotations improve validation performance. Overall, the skip architecture and dense prediction approach achieve strong performance across datasets, with late fusion and multi-task heads effectively handling RGB-D and joint label prediction while retaining efficient inference.

Converting pretrained classification networks into dense fully convolutional models yields usable semantic segmentation, with adapted VGG giving the strongest accuracy and AlexNet the fastest inference. GoogLeNet uses far fewer parameters and a much larger receptive field, yet does not reach VGG-level segmentation accuracy, showing that classification success does not automatically transfer to dense prediction. All compared models share the same coarsest stride, while receptive field size and parameter count vary widely across architectures. FCN-VGG16 leads the compared architectures in mean intersection over union and is described as near state-of-the-art on the validation set. FCN-AlexNet is the fastest of the three but has the lowest mean intersection over union, while FCN-GoogLeNet is more parameter efficient yet still trails FCN-VGG16.

End-to-end training clearly outperforms fine-tuning only the last layer, with the largest gap appearing on region-based measures. Increasing model resolution by reducing the final stride from 32 to 16 and then to 8 provides steady but smaller improvements across all metrics. The best overall results come from the finest stride setting, while fixed-layer training remains substantially weaker. Training only the last layer performs poorly compared with end-to-end fine-tuning. Lower strides improve segmentation quality, but the gain from 16 to 8 is small.

The fully convolutional network achieves the best mean IU on the PASCAL VOC 2011 and 2012 test sets, with about a 20% relative improvement over the previous state of the art. It also uses far less inference time than the compared method, making dense pixel prediction much faster. Highest mean IU on both VOC test sets About 20% relative gain over SDS Inference time drops from tens of seconds to subsecond latency

On NYUDv2, the RGB-only model performs comparably to a prior result, while early-fusion RGB-D input provides little additional benefit. Late fusion of RGB and HHA depth embeddings improves performance more clearly, and the improved late-fusion variant further increases accuracy and IU-style metrics. Early fusion of RGB and depth channels provides little benefit over RGB-only training. Late fusion of RGB and HHA predictions yields the strongest improvements in mean accuracy and mean IU. A less coarse late-fusion model further improves results over the coarser RGB-HHA setting.

The results compare several baseline approaches on SIFT Flow pixel segmentation, showing different strengths across class segmentation metrics. Some methods favor overall pixel accuracy, while others improve mean per-class accuracy, and a transfer method is compared on geometric accuracy. The described fully convolutional model combines semantic and geometric prediction in one joint model and is reported to achieve state-of-the-art performance on both tasks. The baseline results reveal a trade-off between overall pixel accuracy and mean per-class accuracy. The joint model is reported to handle both semantic and geometric prediction while maintaining state-of-the-art task performance.

Experiments convert pretrained classification networks into fully convolutional models and compare their dense prediction behavior, showing that VGG adaptations achieve the strongest segmentation performance while AlexNet is faster and GoogLeNet is more compact but less accurate. Further studies validate that end-to-end training and finer output resolution consistently improve segmentation quality, and the resulting model provides strong performance on standard segmentation benchmarks with much faster inference than prior approaches. Additional evaluations on depth and pixel segmentation tasks indicate that late fusion of depth information and joint semantic-geometric modeling can further improve robustness and overall accuracy.