In this paper, we propose a convolutional layer inspired by optical flowalgorithms to learn motion representations. Our representation flow layer is afully-differentiable layer designed to capture the flow' of any representationchannel within a convolutional neural network for action recognition. Itsparameters for iterative flow optimization are learned in an end-to-end fashiontogether with the other CNN model parameters, maximizing the action recognitionperformance. Furthermore, we newly introduce the concept of learningflow offlow' representations by stacking multiple representation flow layers. Weconducted extensive experimental evaluations, confirming its advantages overprevious recognition models using traditional optical flows in bothcomputational speed and performance. Code/models available here:https://piergiaj.github.io/rep-flow-site/