When building a unified vision system or gradually adding new capabilities toa system, the usual assumption is that training data for all tasks is alwaysavailable. However, as the number of tasks grows, storing and retraining onsuch data becomes infeasible. A new problem arises where we add newcapabilities to a Convolutional Neural Network (CNN), but the training data forits existing capabilities are unavailable. We propose our Learning withoutForgetting method, which uses only new task data to train the network whilepreserving the original capabilities. Our method performs favorably compared tocommonly used feature extraction and fine-tuning adaption techniques andperforms similarly to multitask learning that uses original task data we assumeunavailable. A more surprising observation is that Learning without Forgettingmay be able to replace fine-tuning with similar old and new task datasets forimproved new task performance.