One-sentence Summary

Li and Hoiem from University of Illinois at Urbana-Champaign propose Learning without Forgetting, which uses knowledge distillation from the original network's soft outputs as pseudo-regularization to preserve existing capabilities when training a Convolutional Neural Network on new tasks without requiring access to the original task data.

Key Contributions

• The paper proposes Learning without Forgetting, a method for extending a convolutional neural network with new capabilities when the original training data are unavailable. It trains on new task data while regularizing the network so that outputs for the original tasks remain close to those of the pre-trained network.

• The adaptation procedure jointly optimizes shared representations and the final layer, discouraging changes to old task outputs rather than only freezing task-specific parameters. It also uses a warm-up step before full fine-tuning to improve practical transfer.

• Experiments using ImageNet as the initial network and new tasks such as PASCAL VOC show that Learning without Forgetting preserves original task performance while using only new task data and approaches multitask learning that uses unavailable original data. It also outperforms Less Forgetting Learning on new task performance and can improve over fine-tuning in some settings with similar original and new task datasets.

Introduction

The authors target incremental vision model updates, such as adding new object, safety, or scene classifiers to an existing CNN without retraining from old data. This matters because practical systems need to adapt to new tasks while retaining prior capabilities and avoiding catastrophic forgetting, especially when legacy data is proprietary, unrecorded, or too costly to reuse. Prior methods have limitations: feature extraction preserves old knowledge but often underperforms on new tasks; fine-tuning improves new tasks but changes shared parameters and can degrade old tasks; duplicating fine-tuned networks increases test time; joint training is strong but requires old training data and becomes cumbersome. The authors expand Learning without Forgetting, which uses only new task examples to optimize new task accuracy while distilling the original network’s predictions as soft targets. This preserves old outputs, regularizes the shared representation, and improves both new and prior task performance with lower computational cost and simpler deployment than joint training or multiple specialized networks.

Method

The authors address new-task adaptation with a CNN that contains shared parameters $\theta_s$θs​ and task-specific parameters. The previously learned task is represented by $\theta_o$θo​, while the new task introduces additional parameters $\theta_n$θn​. The method uses only images and labels from the new task, while preserving useful behavior from earlier tasks by constraining the network outputs to remain close to the responses of the original network. The overall structure can be understood as adding a new output branch to an existing model and training both branches with a combined objective. Refer to the framework diagram.

The pipeline begins by passing each new-task training image through the original network, defined by $\theta_s$θs​ and $\theta_o$θo​. For each image, the network produces the old-task response vector ${\bf y}_o$yo​, which consists of label probabilities in the classification setting. These recorded responses are treated as teacher signals during later training. Then, output nodes for the new classes are added to the final shared layer and connected with newly initialized weights $\theta_n$θn​. The number of added parameters is typically small because it equals the number of new classes multiplied by the number of nodes in the last shared layer. The authors use Xavier initialization for the new parameters in their implementation.

Training optimizes a hybrid objective that combines supervised learning on the new task with response preservation on the old task. For the new task, the network output $\hat{\bf y}_n$y^​n​ is a softmax distribution, and ${\bf y}_n$yn​ is the one-hot ground truth label. The new-task loss uses multinomial logistic loss:

If there are multiple new tasks or multi-label outputs, the loss is summed across tasks and labels.

For the old tasks, the method does not require access to the original training images. Instead, it enforces that the current network outputs $\hat{\bf y}_o$y^​o​ match the recorded responses ${\bf y}_o$yo​ from the original network on the new-task images. This is done with a knowledge distillation loss. The probability values are first modified using a temperature $T$T:

The old-task loss is then a temperature-scaled cross-entropy term:

The temperature $T=2$T=2 is used in the reported setup. Values larger than one increase the relative influence of smaller probabilities, encouraging the network to preserve class similarity information rather than only matching the most probable label. The total training objective combines these terms with regularization:

where $\mathcal{R}$R is a simple weight decay term, set to 0.0005 in the experiments, and $\lambda_o$λo​ controls the balance between new-task learning and old-task preservation. A larger $\lambda_o$λo​ gives more weight to the old-task response-preserving loss, which shifts performance toward the original task at the possible expense of the new task. Refer to the tradeoff plot.

Optimization is performed in two stages. In the warm-up stage, the shared parameters $\theta_s$θs​ and old-task parameters $\theta_o$θo​ are frozen, and only the newly initialized parameters $\theta_n$θn​ are trained to convergence. This gives the new head a stable initialization before the shared representation is adapted. In the joint-optimize stage, all parameters $\theta_s$θs​, $\theta_o$θo​, and $\theta_n$θn​ are trained together until convergence. The authors adopt this two-stage scheme for fairness with comparison methods and for a small performance benefit.

In implementation, the networks are trained with stochastic gradient descent using momentum of 0.9. Dropout remains enabled in fully connected layers. Data preprocessing follows the original network setup, including mean subtraction, resizing, and augmentation with random fixed-size crops, random mirroring, and RGB variance perturbation. The learning rate for adaptation is set substantially lower than the original training learning rate, commonly around 0.1 to 0.02 times the original value. The feature extraction baseline uses a much larger learning rate because it optimizes only a small number of parameters. The number of warm-up and joint-optimize epochs is selected on held-out validation data, and the learning rate is reduced by a factor of 10 when validation accuracy plateaus.

The method can be viewed as an approximation to joint training without storing the old dataset. Joint training would use images and labels from both old and new tasks and apply separate task losses to their corresponding images. In contrast, the proposed method uses new-task images as the common input and uses the recorded old-task responses as substitute supervision for old behavior. This removes the need to access or store legacy training data and allows each new-task image to pass through the shared feature extractor once for both new-task loss and old-task distillation loss. The baseline feature extraction variant freezes the shared parameters, disables dropout, and adds a small two-layer network on top of the frozen features, while joint training alternates batches from different tasks and balances the number of images sampled per task in each epoch.

Experiment

Experiments evaluate Learning without Forgetting for adapting pretrained convolutional networks to new image classification tasks while retaining old-task performance, comparing it against feature extraction, fine-tuning, partial fine-tuning, Less Forgetting Learning, and joint training across datasets with varying similarity. LwF generally provides stronger new-task adaptation than fine-tuning-based baselines and preserves old-task performance better than full fine-tuning, often approaching joint training despite not requiring old-task training data, though highly dissimilar or sequentially added tasks can still cause greater performance drift. Ablations show that preserving old-task outputs with a distillation-style loss is more effective than weight regularization, reduced shared learning rate, additional task-specific layers, or network expansion, supporting LwF as a practical and interpretable alternative to standard fine-tuning for continual prediction expansion.

The comparison shows that Learning without Forgetting preserves performance on both new and original tasks while requiring only medium storage and no access to previous task data. It is fast to train and test, offering a more balanced alternative than fine tuning, which is efficient but weak on original tasks, or joint training, which is accurate but slow, storage heavy, and dependent on prior data. Learning without Forgetting achieves strong performance on new tasks while maintaining original task accuracy Joint training performs well but is slower and requires previous task data and larger storage Fine tuning is efficient but can degrade original task performance

The experiments compare continual learning strategies across task performance, efficiency, storage, and data requirements. Learning without Forgetting balances strong performance on new and original tasks while needing only medium storage and no prior task data. This makes it more favorable than fine tuning, which is efficient but weak on original tasks, and joint training, which is accurate but slower, storage heavy, and dependent on previous data.