One-sentence Summary

Microsoft Research's Faster R-CNN introduces a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, enabling nearly cost-free region proposals and achieving 5fps detection with state-of-the-art accuracy on PASCAL VOC benchmarks.

Key Contributions

• Introduces a Region Proposal Network (RPN) that is fully convolutional and predicts objectness scores and bounding-box regressions for multiple scales and aspect ratios at each convolutional feature-map position while sharing full-image convolutional features with the detection network.
• Proposes an alternating optimization scheme that fine-tunes the RPN and Fast R-CNN for proposal generation and object detection, producing a unified detector with shared convolutional features and a marginal proposal cost of about 10 milliseconds per image.
• Demonstrates state-of-the-art object detection accuracy on PASCAL VOC 2007 and 2012, achieving 73.2% and 70.4% mAP with 300 proposals per image and running at about 5 frames per second on a GPU using a VGG-16 detector.

Introduction

Object detection systems based on region proposal methods and region-based convolutional networks enable accurate localization and recognition, but the proposal stage matters because it can dominate overall runtime. Prior methods such as Selective Search and EdgeBoxes rely on engineered features and CPU inference, making them slower than the detector and missing the chance to share computation with the detection network. The authors leverage a Region Proposal Network that reuses the convolutional features of a Fast R-CNN detector, adds fully convolutional heads to predict proposal scores and bounds, and trains the proposal and detection tasks end to end. This makes proposal generation nearly cost free, around 10 ms per image, while improving detection accuracy and enabling practical GPU frame rates.

Method

The authors leverage a Region Proposal Network, or RPN, to replace heuristic proposal generation with a learnable, fully convolutional module. The RPN is designed to share convolutional layers with a downstream Fast R CNN detector, so the backbone feature map produced by the last shared convolutional layer becomes the input to a small proposal network. This mini network slides across the feature map. At each spatial location, an $n \times n$n×n window is mapped to a lower dimensional feature vector, 256 dimensional for the ZF backbone and 512 dimensional for the VGG backbone. The vector then feeds into two sibling $1 \times 1$1×1 convolutional branches: a box regression branch and an object classification branch. Refer to the framework diagram.

At every sliding location, the authors instantiate $k$k anchor boxes. Each anchor is a reference box centered at that sliding window and associated with a predefined scale and aspect ratio. By using three scales and three aspect ratios, the model produces $k = 9$k=9 anchors per location. This design makes the proposal generation translation invariant: the same proposal function is applied at every spatial position, and anchors are relative reference boxes rather than globally fixed regions. For a feature map of size $W \times H$W×H, the total number of anchors is $W H k$WHk.

Each anchor is assigned a binary training label. An anchor is positive if it has the highest Intersection over Union overlap with a ground truth box, or if it overlaps any ground truth box with Intersection over Union higher than 0.7. An anchor is negative if its overlap with all ground truth boxes is lower than 0.3. Anchors with overlap between these thresholds are ignored during training. The authors then minimize a multi task loss over classification and regression terms:

Here, $p_i$pi​ is the predicted probability that anchor $i$i contains an object, and $p_i^*$pi∗​ is the ground truth label, equal to 1 for positive anchors and 0 for negative anchors. The term $t_i$ti​ denotes the four predicted box coordinates, while $t_i^*$ti∗​ denotes the four coordinates of the ground truth box associated with a positive anchor. The classification loss is a log loss over object versus background, and the regression loss uses a robust smooth $L_1$L1​ loss. Because the term $p_i^* L_{reg}$pi∗​Lreg​ is activated only when $p_i^* = 1$pi∗​=1, the regression loss is applied only to positive anchors.

For coordinate regression, the authors learn offsets relative to the anchor box rather than predicting absolute box coordinates directly. The target and prediction are parameterized as:

where $x, y, w, h$x,y,w,h are the predicted box center, width, and height, $x_a, y_a, w_a, h_a$xa​,ya​,wa​,ha​ are the anchor box parameters, and $x^*, y^*, w^*, h^*$x∗,y∗,w∗,h∗ are the ground truth box parameters. Since the RPN extracts features from fixed $n \times n$n×n sliding windows, the authors learn $k$k separate regression branches, one for each anchor scale and aspect ratio. This allows the model to predict boxes of different sizes while still using fixed size spatial features.

Training the RPN is performed end to end with back propagation and stochastic gradient descent. The authors use an image centric sampling strategy, where each mini batch comes from a single image containing both positive and negative anchors. To avoid strong imbalance toward background anchors, they randomly sample 256 anchors per image to compute the loss, with a positive to negative ratio of up to 1:1. If fewer than 128 positive anchors are present, the mini batch is padded with negative anchors. New layers are initialized from a zero mean Gaussian distribution with standard deviation 0.01, while shared convolutional layers are initialized from a pretrained ImageNet model. The learning rate schedule is 0.001 for the first 60k mini batches and 0.0001 for the next 20k mini batches, with momentum 0.9 and weight decay 0.0005. For VGG, the authors tune layers starting from conv3 onward to conserve memory.

To share convolutional features between the proposal generator and the detector, the authors use a pragmatic alternating optimization procedure. In the first stage, the RPN is trained end to end using the region proposal objective. In the second stage, a separate Fast R CNN detector is trained using proposals generated by the RPN, with the detector also initialized from an ImageNet pretrained model. At this point, the two networks do not yet share convolutional weights. In the third stage, the detector weights are used to initialize the RPN, but the shared convolutional layers are frozen and only the RPN specific layers are fine tuned. In the fourth stage, the shared convolutional layers remain fixed while the fully connected layers of the Fast R CNN are fine tuned. After this process, both networks share the same convolutional backbone and form a unified detection system.

For implementation, both the proposal network and the detector operate on single scale images with the shorter image side resized to 600 pixels. The stride of the final shared convolutional layer is 16 pixels on the resized image. The authors use three anchor scales with box areas of $128^2$1282, $256^2$2562, and $512^2$5122 pixels, and three aspect ratios of 1:1, 1:2, and 2:1. Anchors that cross the image boundary are ignored during training to avoid destabilizing optimization, while at test time proposals that extend outside the image are clipped to the image boundary. Because many proposals overlap, the authors apply non maximum suppression based on the classification scores with an Intersection over Union threshold of 0.7, reducing the raw proposals to about 2k candidate regions per image for detection.

Experiment

The authors evaluate on PASCAL VOC using ZF and VGG backbones, primarily measuring detection accuracy and runtime. The experiments show that RPN can match or surpass conventional proposal methods with far fewer proposals and shared convolutional computations, enabling faster detection. Ablations indicate that feature sharing, deeper networks, classification scores, and box regression all improve proposal quality, while recall analysis and comparisons with a one-stage detector support the effectiveness of a two-stage proposal-and-detection cascade.

Fast R-CNN detectors using ZF features show that RPN proposals lead to the strongest detection results among the compared methods. The improvement appears despite RPN using only about 300 proposals, whereas Selective Search and Edge Boxes use about 2,000. Shared convolutional computations also make the RPN based workflow faster and cheaper at test time. RPN with shared features slightly outperforms Selective Search and Edge Boxes in detection mAP. RPN retains good detection quality with far fewer proposals than traditional region proposal methods. Sharing convolutional layers between proposal generation and detection supports faster and lower cost inference.

On the PASCAL VOC 2007 test set with a VGG-16 based Fast R-CNN detector, RPN achieves strong detection quality while using fewer proposals than Selective Search. Sharing convolutional layers between the proposal network and detector improves mean Average Precision and reduces runtime compared with unshared RPN. Training on the combined VOC 2007 and VOC 2012 data gives the best results for both methods. Shared RPN with VGG-16 gives better mean Average Precision than Selective Search. Shared RPN is faster than unshared RPN because convolutional computation is reused. Adding VOC 2012 training data improves mean Average Precision for both proposal methods.

On PASCAL VOC 2012, a shared RPN with VGG-16 improves Fast R-CNN detection over Selective Search while using far fewer proposals. Training on the combined 2007 and 2012 set raises mAP for both approaches, and the shared RPN gives the strongest reported accuracy. The comparison supports learned, shared region proposals as an effective alternative to many hand-generated proposals. Shared RPN with VGG-16 outperforms Selective Search on PASCAL VOC 2012 while using far fewer proposals Using the combined 2007 and 2012 training data improves mAP for both proposal sources Shared feature learning between RPN and Fast R-CNN yields the highest mAP among the reported variants

RPN replaces the slow CPU-based Selective Search proposal step with an in-network proposal stage, sharply reducing total detection time. For the same VGG backbone, RPN lowers proposal and region-wise costs, making the full system much faster. The smaller ZF backbone with RPN is the fastest among the reported configurations. Selective Search dominates the total time in the VGG pipeline because its proposal step runs on CPU. RPN keeps the convolution cost similar on VGG while greatly reducing proposal time. Using ZF instead of VGG with RPN gives the shortest total runtime and highest processing rate among the reported timings.

The two-stage proposal plus detection system with RPN and Fast R-CNN achieves higher detection mAP than the one-stage dense sliding-window variants on PASCAL VOC 2007 with the ZF model. The one-stage system gains little from using multiple image scales, while requiring many more regions to process and therefore running more slowly. Two-stage RPN proposals lead to better detection accuracy than one-stage dense sliding-window proposals. Adding image scales to the one-stage variant provides only a small mAP gain. The two-stage approach uses far fewer regions than the one-stage sliding-window pipeline.

Experiments on PASCAL VOC evaluate Fast R C NN detectors with ZF and VGG backbones, comparing RPN proposals with Selective Search, Edge Boxes, and dense sliding window pipelines. RPN achieves stronger detection quality while using far fewer proposals, and sharing convolutional features between RPN and Fast R C NN improves accuracy and efficiency. The shared design also outperforms one stage variants, supports faster lower cost inference, and benefits from larger combined VOC training data.