DFNDR-12M
收藏资源简介：
# Dataset Card for DFNDR-12M <!-- Provide a quick summary of the dataset. --> This dataset contains synthetic captions, embeddings, and metadata for DFNDR-12M. The metadata has been generated using pretrained image-text models on DFN-12M, a uniformly sampled subset of 12.8M samples from [DFN-2B](https://arxiv.org/abs/2309.17425). For details on how to use the metadata, please visit our [ml-mobileclip repository](https://github.com/apple/ml-mobileclip). For code to generate multi-modal reinforced datasets at large scale see [ml-mobileclip-dr repository](https://github.com/apple/ml-mobileclip). A BFloat16 version of this dataset is available at [apple/DFNDR-12M-bf16](https://huggingface.co/datasets/apple/DFNDR-12M-bf16). **Note that this release does not contain original ground-truth captions. Please refer to [DFN-2B](https://huggingface.co/datasets/apf1/datafilteringnetworks_2b) and [DataComp](https://github.com/mlfoundations/datacomp#downloading-commonpool) for download instructions.** ## Dataset Details ### Dataset Description <!-- Provide a longer summary of what this dataset is. --> DFNDR is an image-text dataset that builds upon the multi-modal dataset reinforcement strategy introduced in MobileCLIP and improved in MobileCLIP2. We create DFNDR-2B by reinforcing [DFN-2B](https://arxiv.org/abs/2309.17425), a 2B filtered subset of DataComp-12B. DFNDR-12M is created by reinforcing DFN-12M, a uniformly sampled subset of 12.8M samples from DFN-2B. Compared to DataCompDR, DFNDR uses an ensemble of two stronger DFN teachers (`DFN2B-CLIP-ViT-L-14` and `DFN2B-CLIP-ViT-L-14-39B`) and improved synthetic captions generated by [`MobileCLIP2-CoCa-ViT-L-14`](https://huggingface.co/apple/mobileclip2_coca_dfn2b_s13b_mscoco38k_s12m_context77). We apply 30 strong random image augmentations for DFNDR-12M (2 for DFNDR-2B). We compute embeddings of the teacher ensemble on augmented images as well as real and synthetic captions. Embeddings are 1536-D concatenations of 2x768-D vectors. One seen sample for DFNDR is a triplet of one randomly augmented image, one ground-truth caption, and one randomly picked synthetic caption. - **Curated by:** Original data by [DataComp](https://www.datacomp.ai/) and metadata by Apple. - **License:** We distribute our metadata under our [license](https://github.com/apple/ml-mobileclip/blob/main/LICENSE_DATA). The original image url-text samples and metadata were released by [DataComp](https://www.datacomp.ai/) under Creative Common CC-BY-4.0 license. The individual images are under their own copyrights. - **Repository:** [ml-mobileclip GitHub](https://github.com/apple/ml-mobileclip) - **Paper:** [MobileCLIP2 paper](https://openreview.net/forum?id=WeF9zolng8) ## Uses <!-- Address questions around how the dataset is intended to be used. --> Training with DFNDR shows significant learning efficiency improvement compared to standard CLIP training. Training on DFNDR-12M is up to 5x more efficient compared with DataComp-1B 12M, 3.3x compared with DFN-12M, and 1.3x compared to DataCompDR-12M. ## Dataset Structure <!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. --> ``` - <uid>.url.txt: Image URL (string) - <uid>.syn.json: - syn_text_dfn_mscoco38k: List of synthetic captions (list[string]) - <uid>.paug.json: - param_aug: List of augmentation parameters (list[list[Union[int,float]]]) - <uid>.npz - image_emb: List of image embeddings for multiple image augmentations (list[list[float]]) - text_emb: List of text embeddings for ground-truth/synthetic captions (list[list[float]]) - syn_text_dfn_mscoco38k_emb: List of embeddings for synthetic captions (list[list[float]]) ``` ## Citation **[MobileCLIP2: Improving Multi-Modal Reinforced Training](http://arxiv.org/abs/2508.20691). (TMLR 2025 Featured)** *Fartash Faghri, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, Alexander T Toshev, Oncel Tuzel, Hadi Pouransari.* ```bibtex @article{faghri2025mobileclip2, title={Mobile{CLIP}2: Improving Multi-Modal Reinforced Training}, author={Fartash Faghri and Pavan Kumar Anasosalu Vasu and Cem Koc and Vaishaal Shankar and Alexander T Toshev and Oncel Tuzel and Hadi Pouransari}, journal={Transactions on Machine Learning Research}, issn={2835-8856}, year={2025}, url={https://openreview.net/forum?id=WeF9zolng8}, note={Featured Certification} } ``` **[MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training](https://arxiv.org/pdf/2311.17049.pdf). (CVPR 2024)** *Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel.* ```bibtex @InProceedings{mobileclip2024, author = {Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel}, title = {MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2024}, } ```
