Previous methods decompose blind super resolution (SR) problem into twosequential steps: \textit{i}) estimating blur kernel from given low-resolution(LR) image and \textit{ii}) restoring SR image based on estimated kernel. Thistwo-step solution involves two independently trained models, which may not bewell compatible with each other. Small estimation error of the first step couldcause severe performance drop of the second one. While on the other hand, thefirst step can only utilize limited information from LR image, which makes itdifficult to predict highly accurate blur kernel. Towards these issues, insteadof considering these two steps separately, we adopt an alternating optimizationalgorithm, which can estimate blur kernel and restore SR image in a singlemodel. Specifically, we design two convolutional neural modules, namely\textit{Restorer} and \textit{Estimator}. \textit{Restorer} restores SR imagebased on predicted kernel, and \textit{Estimator} estimates blur kernel withthe help of restored SR image. We alternate these two modules repeatedly andunfold this process to form an end-to-end trainable network. In this way,\textit{Estimator} utilizes information from both LR and SR images, which makesthe estimation of blur kernel easier. More importantly, \textit{Restorer} istrained with the kernel estimated by \textit{Estimator}, instead ofground-truth kernel, thus \textit{Restorer} could be more tolerant to theestimation error of \textit{Estimator}. Extensive experiments on syntheticdatasets and real-world images show that our model can largely outperformstate-of-the-art methods and produce more visually favorable results at muchhigher speed. The source code is available athttps://github.com/greatlog/DAN.git.