Although lots of progress were made in Text Recognition/OCR in recent years,the task of font recognition is remaining challenging. The main challenge liesin the subtle difference between these similar fonts, which is hard todistinguish. This paper proposes a novel font recognizer with a pluggablemodule solving the font recognition task. The pluggable module hides the mostdiscriminative accessible features and forces the network to consider othercomplicated features to solve the hard examples of similar fonts, called HEBlock. Compared with the available public font recognition systems, ourproposed method does not require any interactions at the inference stage.Extensive experiments demonstrate that HENet achieves encouraging performance,including on character-level dataset Explor_all and word-level dataset AdobeVFR