MemNet: A Persistent Memory Network for Image Restoration

Recently, very deep convolutional neural networks (CNNs) have been attractingconsiderable attention in image restoration. However, as the depth grows, thelong-term dependency problem is rarely realized for these very deep models,which results in the prior states/layers having little influence on thesubsequent ones. Motivated by the fact that human thoughts have persistency, wepropose a very deep persistent memory network (MemNet) that introduces a memoryblock, consisting of a recursive unit and a gate unit, to explicitly minepersistent memory through an adaptive learning process. The recursive unitlearns multi-level representations of the current state under differentreceptive fields. The representations and the outputs from the previous memoryblocks are concatenated and sent to the gate unit, which adaptively controlshow much of the previous states should be reserved, and decides how much of thecurrent state should be stored. We apply MemNet to three image restorationtasks, i.e., image denosing, super-resolution and JPEG deblocking.Comprehensive experiments demonstrate the necessity of the MemNet and itsunanimous superiority on all three tasks over the state of the arts. Code isavailable at https://github.com/tyshiwo/MemNet.