One-sentence Summary

Microsoft Research and academic collaborators introduce COCO, a dataset containing 2.5 million labeled instances across 328k images with per-instance segmentations of common objects in complex everyday scenes, positioning object recognition within broader scene understanding and providing comparative baselines against PASCAL and ImageNet.

Key Contributions

• Introduces a large-scale dataset for object recognition in scene understanding, comprising 328k images and 2.5 million per-instance segmented object labels across 91 everyday object categories. The segmentations support precise object localization and evaluation beyond bounding boxes.

• Presents a statistical analysis comparing the dataset to PASCAL, ImageNet, and SUN. The analysis characterizes dataset scale, object categories, instance counts, and context-rich image collection.

• Reports baseline bounding-box and segmentation detection results using a Deformable Parts Model. These results provide an evaluation reference for detection on the new dataset.

Introduction

In computer vision, scene understanding requires models that can recognize objects, localize them precisely, reason about their context, and handle real-world visual complexity. Existing datasets advanced object classification and detection, but many emphasize iconic object views, isolated objects, or coarse bounding-box annotations, which limits progress on non-canonical perspectives, contextual reasoning, and accurate instance-level spatial understanding. The authors address this with the Microsoft COCO dataset, which collects everyday scene images rich in object interactions and provides instance-level segmentation masks for 91 object categories, with more than 2.5 million labeled object instances across 328,000 images. This supports training and evaluation for object detection and segmentation under realistic conditions, including clutter, occlusion, small objects, and multiple objects per image.

Dataset

• MS COCO is built for detecting and segmenting everyday object instances in natural scenes. The authors start from a broad list of visually recognizable object categories, aim for large numbers of instances per category, and collect images with rich object context rather than isolated canonical object photos.

• The category set begins with a 272 candidate list assembled from PASCAL VOC categories, frequent visual object words, categories from sources such as WordNet, LabelMe, and SUN, and object names elicited from children. Co-authors vote on the candidates by frequency, usefulness, and diversity, then choose 91 final categories balanced across supercategories such as animals, vehicles, and furniture. The final categories are entry-level “thing” categories, not “stuff” categories, and categories where obtaining more than about 5,000 instances is difficult are removed. All PASCAL VOC categories are kept for compatibility.

• The 2014 release is limited to 80 segmented categories. The authors exclude 11 categories from segmentation, including hat, shoe, eyeglasses, mirror, window, door, street sign, plate, desk, blender, and hair brush, because of excessive instance counts, labeling ambiguity, confusion with related categories, or too few instances.

• Candidate images are collected primarily from Flickr to favor non iconic, context-rich photos. Instead of searching for single objects, which often returns centered iconic images, the authors query pairs of object categories, such as dog plus car, and pairs of object and scene categories. They use 40 indoor and outdoor scene categories selected from SUN to support scene object queries.

• Image filtering focuses on diversity and context. At most five photos by the same photographer are downloaded within a short time window, and near-duplicate images are explicitly removed. When rare categories cannot be found with paired queries, single-category searches are used followed by explicit crowd-worker filtering of iconic or invalid images. The resulting pool contains about 328,000 images, with an average of 3.5 object categories and 7.7 object instances per image.

• The dataset is annotated through Amazon Mechanical Turk in three stages: category labeling, instance spotting, and instance segmentation. The category taxonomy is organized into 11 supercategories to speed labeling. For each image, 8 workers indicate whether any category in a supercategory is present, and if present, place a category icon on one instance of that category.

• In the instance spotting stage, 8 workers mark additional instances of each labeled category, up to 10 instances per category per image. The first instance found during category labeling is shown to help workers locate more objects, and a magnifying-glass interface supports small-object labeling.

• In the segmentation stage, each spotted instance is segmented once, after workers pass a category-specific training task. Segmentations are then verified by multiple workers, initially three and up to five if needed. Poor segmentations are rejected and added back to the unsegmented pool, and work from consistently poor workers is discarded.

• For dense groups containing more than about 10 instances of the same category, the authors mark the remaining objects as “crowds” using a single possibly multipart segment. These crowd regions are ignored during evaluation and do not affect detector scores.

• The dataset includes pixel-level instance segmentations for over 2.5 million object instances and five caption annotations per image in MS COCO. The 2014 release contains 82,783 training images, 40,504 validation images, and 40,775 test images. Training and validation images contain about 886,000 segmented object instances, including nearly 270,000 people.

• The planned cumulative 2015 release totals 165,482 training images, 81,208 validation images, and 81,434 test images. The authors split the data to minimize near-duplicate leakage by removing near duplicates and grouping images by photographer and date taken.

• The authors release annotations for training and validation splits, but keep test annotations hidden for evaluation. No specific image cropping recipe or training mixture ratio is described in this dataset section; the main processing details are category selection, image collection and deduplication, hierarchical labeling, instance spotting, segmentation verification, crowd handling, and split construction.

Method

The method is a staged crowdsourcing annotation pipeline rather than a conventional end-to-end model. The authors first construct a non-iconic image pool by querying category pairs or category-scene combinations, then require workers to remove invalid and overly canonical examples. In the filtering interface, instruction specificity and visual examples guide selection, and candidates appear as grids for efficient de-selection. This early filtering step is designed to reduce iconicity bias before annotation begins.

After filtering, each image enters category labeling. Because the category space is large, asking workers to answer many binary questions per image would be inefficient. Instead, the authors group categories into super-categories. A worker first checks whether any member of a super-category appears in the image. If the super-category is present, the worker then identifies the specific subordinate category by dragging its icon onto one visible instance. Only one exemplar per category is required at this stage, and multiple workers label each image to promote recall. False positives are later corrected through additional stages.

The next stage is instance spotting. For each category identified in the previous step, workers mark every visible instance of that category. The pipeline improves efficiency by showing the worker an instance already found in the category-labeling stage, which helps workers quickly orient to the image content. A magnification tool is also provided to support detection of small or hard-to-see objects. Workers place a cross inside each instance region, with a limit on the number of instances annotated per category in extreme cases.

The most expensive stage is instance segmentation. The authors modify an interface for polygonal labeling and provide visual reminders of the target category, along with already completed masks when they exist. Each instance is segmented only once to control cost, but the pipeline compensates with strict worker training and quality control. Before performing productive segmentation, workers complete a category-specific training task and must produce a mask close enough to ground truth before being approved. During segmentation, workers can also indicate that a labeled category is absent or that all remaining instances have already been segmented.

To avoid redundant segmentation, the system processes instance annotations sequentially and ignores instance locations already covered by an existing mask. This prevents different workers from repeatedly segmenting the same object. After segmentation, the pipeline adds verification. Each mask is judged by multiple workers, and poor-quality segmentations are discarded. The rejected instances are returned to the unsegmented pool for reprocessing. Workers whose masks repeatedly fail verification are removed from the approved pool.

The final module handles dense object regions. When too many instances of a category appear in one image, individually delineating each object becomes difficult and inconsistent. After a limited number of instances have been segmented, remaining tightly packed objects are labeled as crowd regions rather than exact instance masks. These crowd regions reduce annotation cost and acknowledge ambiguity in highly grouped instances, while evaluation can ignore those regions. The resulting examples illustrate the precision and contextual variety produced by the pipeline.

Experiment

The experiments compare DPM detectors trained on PASCAL VOC and MS COCO using tight bounding boxes, and separately assess generated segmentation masks only when detections are correct. They show that performance falls markedly on MS COCO, indicating its greater difficulty, while COCO-trained models tend to generalize better back to PASCAL despite category-dependent effects from hard, non-iconic images. The segmentation analysis further demonstrates that detection can be decoupled from mask localization to evaluate fine part boundaries, establishing a simple mask-quality baseline for the dataset.

The comparison shows that MS COCO is substantially more challenging than PASCAL VOC, with detection scores falling for both PASCAL-trained and COCO-trained models when evaluated on MS COCO. Training on MS COCO data gives mixed results on PASCAL VOC, improving some categories while hurting others, which suggests that difficult images can help some object classes but act as noise for others. When tested on MS COCO, the COCO-trained model achieves better average performance than the PASCAL-trained model. MS COCO evaluation yields lower detection performance than PASCAL VOC evaluation. The COCO-trained model outperforms the PASCAL-trained model in a minority of PASCAL categories. Dog, cat, and person categories perform worse when the model is trained on MS COCO. Bus, tv, and horse categories benefit from training on MS COCO when tested on PASCAL VOC.

The experiments compare detection performance on PASCAL VOC and MS COCO using models trained on each dataset. They find that MS COCO is substantially more challenging than PASCAL VOC, since detection scores drop on MS COCO for both trained models, while the COCO-trained model achieves better average performance on COCO. Training on MS COCO has mixed effects on PASCAL VOC, improving some classes such as bus, tv, and horse while hurting others such as dog, cat, and person, suggesting that difficult images help certain object categories but act as noise for others.