A novel ordinal regression algorithm, called moving window regression (MWR),is proposed in this paper. First, we propose the notion of relative rank($\rho$ρ-rank), which is a new order representation scheme for input andreference instances. Second, we develop global and local relative regressors($\rho$ρ-regressors) to predict $\rho$ρ-ranks within entire and specific rankranges, respectively. Third, we refine an initial rank estimate iteratively byselecting two reference instances to form a search window and then estimatingthe $\rho$ρ-rank within the window. Extensive experiments results show that theproposed algorithm achieves the state-of-the-art performances on variousbenchmark datasets for facial age estimation and historical color imageclassification. The codes are available at https://github.com/nhshin-mcl/MWR.