apple/DFNDR-12M

Dataset Card for DFNDR-12M

<!-- Provide a quick summary of the dataset. -->

This dataset contains synthetic captions, embeddings, and metadata for DFNDR-12M. The metadata has been generated using pretrained image-text models on DFN-12M, a uniformly sampled subset of 12.8M samples from DFN-2B. For details on how to use the metadata, please visit our ml-mobileclip repository. For code to generate multi-modal reinforced datasets at large scale see ml-mobileclip-dr repository. A BFloat16 version of this dataset is available at apple/DFNDR-12M-bf16.

Note that this release does not contain original ground-truth captions. Please refer to [DFN-2B](https://huggingface.co/datasets/apf1/datafilteringnetworks_2b) and [DataComp](https://github.com/mlfoundations/datacomp#downloading-commonpool) for download instructions.

Dataset Details

Dataset Description

<!-- Provide a longer summary of what this dataset is. -->

DFNDR is an image-text dataset that builds upon the multi-modal dataset reinforcement strategy introduced in MobileCLIP and improved in MobileCLIP2. We create DFNDR-2B by reinforcing DFN-2B, a 2B filtered subset of DataComp-12B. DFNDR-12M is created by reinforcing DFN-12M, a uniformly sampled subset of 12.8M samples from DFN-2B. Compared to DataCompDR, DFNDR uses an ensemble of two stronger DFN teachers (DFN2B-CLIP-ViT-L-14 and DFN2B-CLIP-ViT-L-14-39B) and improved synthetic captions generated by `MobileCLIP2-CoCa-ViT-L-14`. We apply 30 strong random image augmentations for DFNDR-12M (2 for DFNDR-2B). We compute embeddings of the teacher ensemble on augmented images as well as real and synthetic captions. Embeddings are 1536-D concatenations of 2x768-D vectors. One seen sample for DFNDR is a triplet of one randomly augmented image, one ground-truth caption, and one randomly picked synthetic caption.

• —Curated by: Original data by DataComp and metadata by Apple.
• —License: We distribute our metadata under our license. The original image url-text samples and metadata were released by DataComp under Creative Common CC-BY-4.0 license. The individual images are under their own copyrights.
• —Repository: ml-mobileclip GitHub
• —Paper: MobileCLIP2 paper

Uses

<!-- Address questions around how the dataset is intended to be used. -->

Training with DFNDR shows significant learning efficiency improvement compared to standard CLIP training. Training on DFNDR-12M is up to 5x more efficient compared with DataComp-1B 12M, 3.3x compared with DFN-12M, and 1.3x compared to DataCompDR-12M.

Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

- <uid>.url.txt: Image URL (string)
- <uid>.syn.json:
  - syn_text_dfn_mscoco38k: List of synthetic captions (list[string])
- <uid>.paug.json:
  - param_aug: List of augmentation parameters (list[list[Union[int,float]]])
- <uid>.npz
  - image_emb: List of image embeddings for multiple image augmentations (list[list[float]])
  - text_emb: List of text embeddings for ground-truth/synthetic captions (list[list[float]])
  - syn_text_dfn_mscoco38k_emb: List of embeddings for synthetic captions (list[list[float]])

Citation

[MobileCLIP2: Improving Multi-Modal Reinforced Training](http://arxiv.org/abs/2508.20691). (TMLR 2025 Featured) Fartash Faghri, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, Alexander T Toshev, Oncel Tuzel, Hadi Pouransari.

@article{faghri2025mobileclip2,
  title={Mobile{CLIP}2: Improving Multi-Modal Reinforced Training},
  author={Fartash Faghri and Pavan Kumar Anasosalu Vasu and Cem Koc and
  Vaishaal Shankar and Alexander T Toshev and Oncel Tuzel and Hadi
  Pouransari},
  journal={Transactions on Machine Learning Research},
  issn={2835-8856},
  year={2025},
  url={https://openreview.net/forum?id=WeF9zolng8},
  note={Featured Certification}
}

[MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training](https://arxiv.org/pdf/2311.17049.pdf). (CVPR 2024) Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel.

@InProceedings{mobileclip2024,
  author = {Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel},
  title = {MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2024},
}