DataCompDR-12M is an image-text dataset released by Apple in 2024, containing synthetic text, image embeddings, and metadata. The related paper is "MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training", which aims to improve the training efficiency and performance of image-text models through multimodal data augmentation strategies.

The dataset includes metadata for 12.8 million images, generated from a subset of DataComp-1B. It utilizes the coca_ViT-L-14 model from OpenCLIP to generate 5 synthetic texts for each image, combined with strong random image augmentation techniques. The dataset includes image embeddings, text embeddings, parametric augmentation information, as well as unique image identifiers and hash values, making it suitable for training efficient image-text alignment models.

Dataset Composition

• url.txt: Image URLs (string type)
• syn.json: Contains a list of synthetic texts (syn_text), which are alternative descriptions generated by the model
• paug.json: Contains a list of parametric augmentations (param_aug), recording the specific parameters of image augmentation
• npz: Contains image embeddings (image_emb) and text embeddings (text_emb), both as lists of floating-point numbers
• json: Contains unique image identifiers (uid) and SHA256 hash values of images (sha256)

Each data sample consists of a triplet comprising a randomly augmented image, the original text, and a randomly selected synthetic text. The embedding vectors are 1,536-dimensional, formed by concatenating 768-dimensional vectors from two strong teacher models.

Citation

@InProceedings{mobileclip2024,
  author = {Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel},
  title = {MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2024},
}