apple/DataCompDR-1B

1---2license: apple-amlr3license_name: apple-ascl4license_link: https://github.com/apple/ml-mobileclip/blob/main/LICENSE_DATA5dataset_info:6  features:7  - name: url.txt8    dtype: string9  - name: syn.json10    struct:11    - name: syn_text12      list:13        dtype: string14  - name: paug.json15    struct:16    - name: param_aug17      dtype: string18  - name: npz19    struct:20    - name: image_emb21      list:22        list: float3223    - name: text_emb24      list:25        list: float3226  - name: json27    struct:28    - name: uid29      dtype: string30    - name: sha25631      dtype: string32task_categories:33- text-to-image34- image-to-text35language:36- en37pretty_name: DataCompDR-1B38size_categories:39- 1B<n<10B40---41 42# Dataset Card for DataCompDR-1B43 44<!-- Provide a quick summary of the dataset. -->45 46This dataset contains synthetic captions, embeddings, and metadata for DataCompDR-1B.47The metadata has been generated using pretrained image-text models on [DataComp-1B](https://huggingface.co/datasets/mlfoundations/datacomp_1b).48For details on how to use the metadata, please visit our [github repository](https://github.com/apple/ml-mobileclip).49 50## Dataset Details51 52### Dataset Description53 54<!-- Provide a longer summary of what this dataset is. -->55 56DataCompDR is an image-text dataset and an enhancement to the DataComp dataset.57We reinforce the DataComp dataset using our multi-modal dataset reinforcement strategy.58In particular, we create DataCompDR-1B and DataCompDR-12M by reinforcing the DataComp-1B (BestPool filtering) and a uniform subset of 12.8M samples, DataCompDR-12M.59We have a one-time generation process, the cost of which is amortized over multiple architectures and extensive ablations.60We generate 5 synthetic captions per image using the `coca_ViT-L-14` model in OpenCLIP, and strong random image augmentations (10 for DataCompDR-1B and 30 for DataCompDR-12M).61We compute embeddings of an ensemble of two strong teachers (`ViT-L-14` with pretrained weights `datacomp_xl_s13b_b90k` and openai in OpenCLIP) on augmented images as well as real and synthetic captions.62Embeddings are 1536-D concatenations of 2x768-D vectors.63One seen sample for DataCompDR is a triplet of one randomly augmented image, one ground-truth caption, and one randomly picked synthetic caption.64 65- **Curated by:** Original data by [DataComp](https://www.datacomp.ai/) and metadata by Apple.66- **License:** We distribute our metadata under our [license](https://github.com/apple/ml-mobileclip/blob/main/LICENSE). The original image url-text samples and metadata were released by [DataComp](https://www.datacomp.ai/) under Creative Common CC-BY-4.0 license. The individual images are under their own copyrights.67- **Repository:** [ml-mobileclip GitHub](https://github.com/apple/ml-mobileclip)68- **Paper:** [MobileCLIP paper](https://arxiv.org/abs/2311.17049)69- **Demo:** Coming Soon70 71## Uses72 73<!-- Address questions around how the dataset is intended to be used. -->74 75Training with DataCompDR shows significant learning efficiency improvement compared to the standard CLIP training.76For example, with a single node of 8×A100 GPUs, we achieve 61.7% zero-shot classification on ImageNet-val in approximately one day when training a ViT-B/16 based CLIP from scratch on DataCompDR-12M.77Training with DataCompDR-1B sets new state-of-the-art performance on several metrics (Fig. 2) while still using a fraction of the training compute budget compared to previous works.78Using DataCompDR, we demonstrate 10x-1000x learning efficiency in comparison to DataComp.79 80## Dataset Structure81 82<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->83 84```85- <uid>.url.txt: Image URL (string)86- <uid>.syn.json:87  - syn_text: List of synthetic captions (list[string])88- <uid>.paug.json:89  - param_aug: List of augmentation parameters (list[list[Union[int,float]]])90- <uid>.npz91  - image_emb: List of image embeddings for multiple image augmentations (list[list[float]])92  - text_emb: List of text embeddings for ground-truth/synthetic captions (list[list[float]])93- <uid>.json94  - uid: UID of image-text sample in DataComp (string)95  - sha256: SHA256 hash of the image (string)96```97 98 99## Citation100 101**[MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training](https://arxiv.org/pdf/2311.17049.pdf). (CVPR 2024)**102*Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel.*103 104```bibtex105@InProceedings{mobileclip2024,106  author = {Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel},107  title = {MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training},108  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},109  month = {June},110  year = {2024},111}112```113