Coco Person Faceswap Dataset (COCO-PFS)

COCO Person Faceswap Dataset (COCO-PFS) is a dataset for training and evaluating methods for person-in-context retrieval, which is a task consisting of finding persons in specific contexts, given a natural language text comprising the name of the person as a query (e.g., "President Joe Biden taking a photo with a journalist after his speech").

Specifically, it consists of COCO 2014 images containing persons whose faces have been face-swapped with some identities from VGGFace2 using deepfake tools.

These files are meant to be employed in conjunction with our method IdCLIP, which presents a baseline and some performance study of CLIP-based retrieval methodologies on this challenging benchmark. Please refer to this repo for how to employ this dataset to train and validate models on this dataset.

Files

features.zip contains the features extracted from the faces of the person contained in the respective images, extracted as explained in our paper by employing an MTCNN for detection and an Inception Resnet (V1) for feature extraction, both trained on VGGFace2 and Casia-Webface datasets.

images.zip contains the RGB images obtained by substituting the faces in the COCO 2014 dataset with the ones from VGGFace2 using the roop tool.

Notice that modified captions are not part of the dataset, since they are modified on-the-fly during the dataloading procedure used both at train and test time of IdCLIP. In the near future, we will dump these captions on file and add them to the dataset to make it more self-contained.

Disclaimer and Restrictions

The files from this dataset are released in restricted mode to address several critical concerns:

Privacy: Ensuring the privacy of individuals whose images are included in the dataset is paramount. Modifying faces using deepfake technology raises significant privacy issues.

Deepfake Concerns: The use of deepfake technology can lead to misuse, including identity theft and misinformation. Restricting access helps mitigate these risks.

Licensing Compliance: The dataset incorporates images from COCO and VGGFace2, both of which have specific non-commercial and usage restrictions. Releasing the dataset under request ensures adherence to these licensing terms.

In light of the above, the dataset is only available for scientific and research purposes.