Local windows are routinely used in computer vision and almost withoutexception the center of the window is aligned with the pixels being processed.We show that this conventional wisdom is not universally applicable. When apixel is on an edge, placing the center of the window on the pixel is one ofthe fundamental reasons that cause many filtering algorithms to blur the edges.Based on this insight, we propose a new Side Window Filtering (SWF) techniquewhich aligns the window's side or corner with the pixel being processed. TheSWF technique is surprisingly simple yet theoretically rooted and veryeffective in practice. We show that many traditional linear and nonlinearfilters can be easily implemented under the SWF framework. Extensive analysisand experiments show that implementing the SWF principle can significantlyimprove their edge preserving capabilities and achieve state of the artperformances in applications such as image smoothing, denoising, enhancement,structure-preserving texture-removing, mutual-structure extraction, and HDRtone mapping. In addition to image filtering, we further show that the SWFprinciple can be extended to other applications involving the use of a localwindow. Using colorization by optimization as an example, we demonstrate thatimplementing the SWF principle can effectively prevent artifacts such as colorleakage associated with the conventional implementation. Given the ubiquity ofwindow based operations in computer vision, the new SWF technique is likely tobenefit many more applications.