Makeup transfer is the task of applying on a source face the makeup stylefrom a reference image. Real-life makeups are diverse and wild, which cover notonly color-changing but also patterns, such as stickers, blushes, andjewelries. However, existing works overlooked the latter components andconfined makeup transfer to color manipulation, focusing only on light makeupstyles. In this work, we propose a holistic makeup transfer framework that canhandle all the mentioned makeup components. It consists of an improved colortransfer branch and a novel pattern transfer branch to learn all makeupproperties, including color, shape, texture, and location. To train andevaluate such a system, we also introduce new makeup datasets for real andsynthetic extreme makeup. Experimental results show that our framework achievesthe state of the art performance on both light and extreme makeup styles. Codeis available at https://github.com/VinAIResearch/CPM.