Commonly Uncommon: Semantic Sparsity in Situation Recognition

Semantic sparsity is a common challenge in structured visual classificationproblems; when the output space is complex, the vast majority of the possiblepredictions are rarely, if ever, seen in the training set. This paper studiessemantic sparsity in situation recognition, the task of producing structuredsummaries of what is happening in images, including activities, objects and theroles objects play within the activity. For this problem, we find empiricallythat most object-role combinations are rare, and current state-of-the-artmodels significantly underperform in this sparse data regime. We avoid manysuch errors by (1) introducing a novel tensor composition function that learnsto share examples across role-noun combinations and (2) semantically augmentingour training data with automatically gathered examples of rarely observedoutputs using web data. When integrated within a complete CRF-based structuredprediction model, the tensor-based approach outperforms existing state of theart by a relative improvement of 2.11% and 4.40% on top-5 verb and noun-roleaccuracy, respectively. Adding 5 million images with our semantic augmentationtechniques gives further relative improvements of 6.23% and 9.57% on top-5 verband noun-role accuracy.