What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis

Many new proposals for scene text recognition (STR) models have beenintroduced in recent years. While each claim to have pushed the boundary of thetechnology, a holistic and fair comparison has been largely missing in thefield due to the inconsistent choices of training and evaluation datasets. Thispaper addresses this difficulty with three major contributions. First, weexamine the inconsistencies of training and evaluation datasets, and theperformance gap results from inconsistencies. Second, we introduce a unifiedfour-stage STR framework that most existing STR models fit into. Using thisframework allows for the extensive evaluation of previously proposed STRmodules and the discovery of previously unexplored module combinations. Third,we analyze the module-wise contributions to performance in terms of accuracy,speed, and memory demand, under one consistent set of training and evaluationdatasets. Such analyses clean up the hindrance on the current comparisons tounderstand the performance gain of the existing modules.