One-sentence Summary

Microsoft Research proposes PReLU, which generalizes the traditional rectified activation with a learnable slope, and pairs it with a robust initialization method tailored for rectifier nonlinearities to train extremely deep rectified networks from scratch, achieving $4.94\%$4.94% top-5 error on ImageNet classification and becoming the first result to surpass reported human-level performance on this dataset.

Key Contributions

• The paper introduces PReLU, a learnable generalization of ReLU that adaptively tunes rectifier parameters to improve model fitting with nearly zero extra computational cost and little overfitting risk.

• A robust initialization method is derived by explicitly modeling the nonlinearities and asymmetric responses of rectifier activations, enabling direct training of extremely deep rectified networks, including models with 30 weight layers, from scratch.

• On the ImageNet 2012 classification dataset, the single-model network reaches 5.71% top-5 error, and the multi-model ensemble reaches 4.94% top-5 error, a 26% relative improvement over GoogLeNet and the first reported result surpassing the 5.1% human-level performance.

Introduction

Convolutional neural networks have become important for visual recognition, with applications such as traffic-sign, face, and handwritten-digit recognition, and recent models can even approach or exceed human performance on large image classification tasks like ImageNet. This progress has largely come from larger and more expressive networks combined with regularization, data augmentation, and large datasets. However, many modern networks rely heavily on ReLU activations, whose asymmetric output properties are not well addressed in prior design and training guidance, making very deep rectified networks harder to train stably from scratch. The authors address this by proposing PReLU, an adaptive extension of ReLU that improves accuracy at low cost, and by deriving an initialization method based on the nonlinearity of ReLU/PReLU to help train very deep networks directly from scratch.

Method

The authors structure the method around two tightly coupled modules: an adaptive rectifier activation unit and a weight initialization scheme that respects the statistical behavior of rectifiers. These modules are incorporated into a deep convolutional architecture equipped with spatial pyramid pooling and a practical multi-GPU training procedure. The overall design goal is to make very deep rectified networks easier to optimize while keeping the added computational and parameter overhead minimal.

The first key component is the Parametric Rectified Linear Unit, or PReLU, which generalizes the fixed ReLU by learning the slope of the negative part of the activation. For an input value $y_i$yi​ on the $i$ith channel, the activation is defined as

An equivalent expression is

Here $a_i$ai​ is a learnable coefficient that controls the negative slope. When $a_i=0$ai​=0, the unit reduces to standard ReLU. When $a_i$ai​ is trainable, the model can adapt the activation shape to the data and to the role of each channel or layer. Refer to the shapes in the figure below.

The authors consider two variants. In the channel wise variant, each channel has its own coefficient $a_i$ai​, so the activation shape can vary across feature maps. In the channel shared variant, one coefficient $a$a is shared by all channels in a layer:

This second variant introduces only one extra parameter per layer. The authors argue that even the channel wise version adds very few parameters because the number of new coefficients is proportional to the number of channels, which is tiny compared with the total number of convolutional weights. This design allows the network to learn more specialized nonlinearities without substantially increasing model capacity or risk of overfitting.

Training PReLU is done jointly with the rest of the network through backpropagation. For one layer, the gradient of the loss $\mathcal{E}$E with respect to a coefficient $a_i$ai​ is derived from the chain rule:

where the summation runs over all positions of the feature map. The derivative of the activation with respect to the coefficient is

For the channel shared variant, the gradient is obtained by summing over all channels in the layer. The authors update the coefficients using momentum:

where $\mu$μ is the momentum and $\epsilon$ϵ is the learning rate. They avoid applying weight decay to the PReLU coefficients because regularization would bias $a_i$ai​ toward zero and push the unit back toward ReLU. In their design, PReLU is initialized with a small but nonzero negative slope, typically $a_i=0.25$ai​=0.25, and then learned freely during training.

A central motivation for PReLU comes from the asymmetric behavior of rectifiers. ReLU outputs only nonnegative values, which creates a positive mean response after activation. PReLU can partially offset this bias by allowing a controlled negative response. This effect is reflected in the per-layer mean responses of trained models, where PReLU generally produces smaller mean activations than ReLU. The learned activation therefore behaves less like a fixed hard threshold and more like a data adaptive nonlinearity.

The learned rectifier also changes the optimization trajectory. Because the negative slope is trainable, gradients from inactive or near inactive units can still contribute to updating the activation shape and the preceding layers. This makes the first-order stochastic gradient procedure closer, in an empirical sense, to better conditioned second-order optimization. The practical effect is visible in training curves, where PReLU reduces the validation error more quickly than ReLU under the same optimization schedule.

The second major methodological component is the initialization rule for rectifier networks. Standard weight initialization schemes often assume approximately linear activations and preserve variance under linear propagation. That assumption is invalid for ReLU and PReLU because rectification halves the effective signal variance. The authors derive an initialization rule by tracking the variance of both forward activations and backward gradients.

For a convolutional layer, the preactivation response is

Let $n_l=k^2 c_l$nl​=k2cl​ denote the number of incoming connections for one response, where $k$k is the spatial filter size and $c_l$cl​ is the number of input channels. The input to layer $l$l is the activated output of the previous layer:

If the weights are initialized with zero mean and independent elements, and the inputs are assumed independent, then

Because the activation is a rectifier rather than a linear function, the variance of the activated signal is not the same as the variance of the preactivation. For ReLU, if the preactivation is approximately symmetric around zero, then

which gives

Across many layers, the total forward propagation factor is therefore a product of terms of the form $\frac{1}{2} n_l \operatorname{Var}[w_l]$21​nl​Var[wl​]. To avoid exponential signal shrinkage or explosion, the authors impose the condition

for each rectified layer. This corresponds to initializing weights from a zero mean Gaussian distribution with standard deviation

This scaling explicitly accounts for the fact that ReLU discards, on average, one half of the variance from the negative half space.

The authors repeat the same logic for backpropagation. The gradient propagated backward through a convolutional layer depends on the transposed weight structure. Let $\hat{n}_l=k^2 d_l$n^l​=k2dl​ denote the number of outgoing connections, where $d_l$dl​ is the number of filters in layer $l$l. Under similar independence assumptions, the backward variance relationship becomes

To keep gradients from vanishing or exploding with depth, the backward condition is

This leads to an alternative zero mean Gaussian initialization with standard deviation

For PReLU, the variance factor is modified by the expected magnitude of the negative slope. If the initial coefficient is $a$a, the forward condition becomes

and the backward condition becomes

When $a=0$a=0, these reduce to the ReLU cases. When $a=1$a=1, the initialization approaches the standard linear activation condition. This formulation makes the initialization rule dependent on the actual rectifier family rather than treating the network as linear at initialization.

The resulting initialization design is particularly important for extremely deep rectified networks. Standard linear-activation based initialization can cause very deep models to stall because the effective signal or gradient is multiplied by too many small factors. By contrast, the rectifier-aware scaling stabilizes variance propagation and allows the authors to train deep architectures directly from scratch. The difference is especially visible when comparing convergence behavior in very deep models.

These methodological components are embedded in a convolutional network design intended for high capacity image classification. The baseline architecture uses a deeper and more compact arrangement of convolutional stages compared with a conventional 19 layer network. The first layer uses a larger $7\times 7$7×7 filter with stride two. Additional convolutional layers are placed on smaller feature maps rather than on the largest early feature maps, which keeps computational complexity comparable while reducing the wall clock cost of operating on large spatial resolutions.

Before the first fully connected layer, the network uses spatial pyramid pooling. The pyramid has four levels, with pooling bin sizes of $7\times 7$7×7, $3\times 3$3×3, $2\times 2$2×2, and $1\times 1$1×1, yielding a fixed-length representation from variable spatial extents. This allows the network to combine local and global context without requiring a fixed crop size for testing. The authors further construct deeper and wider variants by adding convolutional layers or increasing filter counts, while retaining the same general design principle of using rectifier units and the proposed initialization.

The training pipeline is deliberately direct. Instead of pretraining a shallower model and then transferring weights to a deeper model, the authors train the full deep architecture from scratch using the proposed initialization. The training process relies on aggressive data augmentation, including random cropping, random scale jittering applied throughout training rather than only during fine tuning, horizontal flipping, and color alteration. This augmentation strategy supports stable optimization of large networks with many parameters.

The optimization procedure uses momentum based stochastic gradient descent. The weight decay is small, momentum is set to a high value, dropout is applied in the first two fully connected layers, and the mini batch size is fixed. The learning rate is reduced in stages as validation error plateaus. Because PReLU coefficients are part of the model, they are optimized during the same forward and backward passes without requiring a separate training stage.

For inference, the architecture uses a dense sliding window testing procedure. The convolutional layers are applied to a resized full image, producing a feature map. Spatial pyramid pooling is then applied to many windows on that feature map, and the fully connected layers compute class scores for those pooled regions. Scores from multiple spatial locations and multiple image scales are averaged, and horizontally flipped images are also used to aggregate predictions. This design turns the network into a dense evaluation system that exploits convolutional feature maps efficiently.

The implementation also includes a simple but effective multi-GPU parallelization strategy. Convolutional layers are parallelized across GPUs through data parallelism. Before the first fully connected layer, the GPUs are synchronized. The fully connected layers are then executed on a single GPU because their computational cost is relatively small compared with the convolutional stack. This choice keeps the implementation simpler than model parallelism while still allowing deep convolutional networks to be trained efficiently with a fixed mini batch size.

Experiment

Experiments on ImageNet classification and transfer to PASCAL VOC detection evaluate how learned PReLU activations and robust weight initialization affect deep convolutional networks. The results show that PReLU consistently improves optimization behavior and generalization over ReLU while preserving learned feature quality, whereas fixed leaky slopes are less effective. The initialization method enables very deep rectifier networks to train reliably, although increasing depth alone does not necessarily improve ImageNet accuracy once models are already sufficiently deep. Overall, deeper and wider rectified models trained end-to-end with learned activations and principled initialization achieve strong classification and transfer performance, with leading multi-model systems surpassing reported human-level accuracy on the benchmark.

The context describes a compact 14 layer network whose layers use 7 by 7 and 2 by 2 convolutions, with learned PReLU coefficients listed for each layer. The learned negative slopes are not zero, showing that PReLU assigns an active slope to negative inputs instead of discarding them like ReLU. Channelwise averages also differ from shared slopes, indicating that activation shape can adapt by layer and by channel. The architecture begins with a large 7 by 7 stride 2 convolution and then uses repeated 2 by 2 convolutional layers with 128 filters. Learned PReLU coefficients vary by layer, with the first and later layers showing larger negative slopes than the middle convolution blocks. The surrounding experiments state that PReLU improves classification and transfer performance over ReLU, consistent with learned nonzero slopes.

The results compare fixed and learned negative-side activation slopes on a small ImageNet classifier. Learned PReLU variants achieve lower top-1 and top-5 errors than ReLU and a fixed leaky ReLU, with channel-wise slopes performing best. The gain comes at almost no extra computational cost. PReLU improves accuracy over ReLU on both top-1 and top-5 metrics. Channel-wise PReLU is slightly better than channel-shared PReLU. A fixed leaky slope gives little improvement over ReLU.

The architectures compare VGG-19 with models A, B, and C, all built from convolutional blocks and pooling stages for a 224 input. Model A shifts convolutions to smaller feature maps to speed up execution while keeping overall cost near VGG-19, and model B adds depth without greatly changing width. Model C keeps the deeper structure of B but widens the network, which greatly increases computational cost and yields better accuracy. Model A shifts workloads to smaller feature maps to improve running speed while keeping complexity close to VGG-19. Adding depth to model A gives model B, but the deeper model performs comparably to the shallower model. Widening model B into model C substantially increases complexity and improves accuracy.

For the large model evaluated on ImageNet 2012 dense testing, PReLU consistently achieves lower top-1 and top-5 error rates than ReLU at each tested input scale. When predictions are combined across multiple scales, PReLU provides a modest but clear reduction in both error measures. The comparison also indicates that PReLU converges faster and maintains lower training and validation errors throughout training. PReLU improves both top-1 and top-5 performance over ReLU on the large model. The accuracy gain is observed across single-scale and multi-scale evaluations. The benefit comes with almost no additional computational cost.

The single-model ImageNet validation results show that PReLU consistently lowers top-1 and top-5 error compared with ReLU for the same architecture. Among the PReLU models, the widest variant achieves the lowest errors, while two models of similar depth perform close to one another. This suggests that increasing depth alone gives limited gains, whereas added width can still improve accuracy. PReLU gives lower top-1 and top-5 error than ReLU in a comparable single-model setting. The widest PReLU model produces the strongest single-model result in the comparison. Widening the model improves accuracy further when the model is already sufficiently deep.

Experiments on compact and large ImageNet networks show that learning negative activation slopes in PReLU consistently improves classification accuracy over ReLU and fixed Leaky ReLU, with channel-wise adaptation performing best at negligible computational overhead. Architectural ablations reveal that simply adding depth yields limited gains, while increased width improves accuracy, especially in already deep models. Together, the results validate PReLU’s adaptive activation shape as an effective, low-cost improvement to deep convolutional networks.