OwensLab/CommunityForensics-Eval

--- license: cc-by-nc-sa-4.0 configs: - config_name: default data_files: - split: CompEval path: data/CompEval-* dataset_info: features: - name: image_name dtype: string - name: format dtype: string - name: resolution list: int64 - name: mode dtype: string - name: image_data dtype: binary - name: model_name dtype: string - name: nsfw_flag dtype: string - name: prompt dtype: string - name: real_source dtype: string - name: subset dtype: string - name: split dtype: string - name: label dtype: int64 - name: architecture dtype: string splits: - name: CompEval num_bytes: 206286046018 num_examples: 51836 download_size: 206224324595 dataset_size: 206286046018 --- # *Community Forensics: Using Thousands of Generators to Train Fake Image Detectors (CVPR 2025)* [Paper](https://arxiv.org/abs/2411.04125) / [Project Page](https://jespark.net/projects/2024/community_forensics/) / [Code (GitHub)](https://github.com/JeongsooP/Community-Forensics) This repository contains the "Comprehensive" evaluation set of the [Community Forensics dataset](https://huggingface.co/datasets/OwensLab/CommunityForensics). This evaluation set contains 21 generative models paired 'real' datasets, which includes [RAISE](https://loki.disi.unitn.it/RAISE/), [COCO](https://cocodataset.org/), [FFHQ](https://github.com/NVlabs/ffhq-dataset), and [LAION](https://laion.ai/). Please note that we distribute this evaluation set for non-commercial research and educational purposes only. If you use this evaluation set, please also cite the aforementioned datasets. Citation information is provided at the end of this page. This is a faithful recreation of the evaluation set used in the paper, with minor modifications for easier redistribution. The evaluation results may vary slightly, but should be very similar. Below is the comparison between the set used in the paper vs. this set: | Classifier | mAP (Paper) | mAP (This repo) | mAcc (Paper) | mAcc (This repo) | | :--- | :--- | :--- | :--- | :--- | | Ours-384 (High res.) | 0.987 | 0.987 | 0.892 | 0.893 | | Ours-224 | 0.971 | 0.972 | 0.861 | 0.860 | | Wang et al. | 0.537 | 0.543 | 0.513 | 0.515 | | Ojha et al. | 0.592 | 0.583 | 0.540 | 0.540 | | GenImage | 0.912 | 0.911 | 0.818 | 0.819 | | RED140 | 0.764 | 0.767 | 0.562 | 0.562 | Note that we compute the mAP/mAcc by averaging the AP/Acc of each generator. ### Train/Test split - To avoid data contamination, please use the attached FFHQ train/test split if you intend to use this evaluation set. [FFHQ train split](https://huggingface.co/datasets/OwensLab/CommunityForensics-Eval/blob/updating_v1/other/ffhq_commfor_train_split.csv)/[test split](https://huggingface.co/datasets/OwensLab/CommunityForensics-Eval/blob/updating_v1/other/ffhq_commfor_test_split.csv). Also, make sure to only train on the "train" split of the COCO dataset. - LAION training distribution is provided [in this link](https://huggingface.co/datasets/OwensLab/CommunityForensics/blob/main/data/Real/laion_commfor_train_subset_2M.csv). - RAISE dataset should not be used during training. # Dataset Structure ## Data Instances Our dataset is formatted in a Parquet data frame of the following structure: ``` { "image_name": "00000162.png", "format": "PNG", "resolution": "[512, 512]", "mode": "RGB", "image_data": "b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\..." "model_name": "stabilityai/stable-diffusion-2", "nsfw_flag": False, "prompt": "montreal grand prix 2018 von icrdesigns", "real_source": "LAION", "subset": "Systematic", "split": "train", "label": "1", "architecture": "LatDiff" } ``` ## Data Fields `image_name`: Filename of an image. \ `format`: PIL image format. \ `resolution`: Image resolution. \ `mode`: PIL image mode (e.g., RGB) \ `image_data`: Image data in byte format. Can be read using Python's BytesIO. \ `model_name`: Name of the model used to sample this image. Has format {author_name}/{model_name} for `Systematic` subset, and {model_name} for other subsets. \ `nsfw_flag`: NSFW flag determined using [Stable Diffusion Safety Checker](https://huggingface.co/CompVis/stable-diffusion-safety-checker). \ `prompt`: Input prompt (if exists). \ `real_source`: Paired real dataset(s) that was used to source the prompts or to train the generators. \ `subset`: Denotes which subset the image belongs to (Systematic: Hugging Face models, Manual: manually downloaded models, Commercial: commercial models). \ `split`: Train/test split. \ `label`: Fake/Real labe l. (1: Fake, 0: Real) `architecture`: Architecture of the generative model that is used to generate this image. (Categories: `LatDiff`, `PixDiff`, `GAN`, `other`, `Commercial`, `real`) - Additional metadata such as model architecture, hyperparameters, and Hugging Face pipeline used can be found under [data/metadata](https://huggingface.co/datasets/OwensLab/CommunityForensics/tree/main/data/metadata). ## Data splits `CompEval` (51,836 images): Comprehensive evaluation set which contains generated images paired with RAISE, COCO, FFHQ, and LAION dataset. This set faithfully reproduces the results of the evaluation set used in our paper. ## Usage examples Default train/eval settings: ```python import datasets as ds import PIL.Image as Image import io commfor_eval = ds.load_dataset("OwensLab/CommunityForensics-Eval", split="CompEval", cache_dir="~/.cache/huggingface/datasets") for i, data in enumerate(commfor_eval): img, label = Image.open(io.BytesIO(data['image_data'])), data['label'] ## Your operations here ## # e.g., img_torch = torchvision.transforms.functional.pil_to_tensor(img) ``` It is also possible to use streaming for some use cases (e.g., downloading only a certain subset or a small portion of data). ```python import datasets as ds import PIL.Image as Image import io # streaming only the evaluation set commfor_eval_stream = ds.load_dataset("OwensLab/CommunityForensics-Eval", split='CompEval', streaming=True) # optionally shuffle the streaming dataset commfor_eval_stream = commfor_sys_stream.shuffle(seed=123, buffer_size=3000) # usage example for i, data in enumerate(commfor_eval_stream): if i>=10000: # use only first 10000 samples break img, label = Image.open(io.BytesIO(data['image_data'])), data['label'] ## Your operations here ## # e.g., img_torch = torchvision.transforms.functional.pil_to_tensor(img) ``` Please check [Hugging Face documentation](https://huggingface.co/docs/datasets/v3.5.0/loading#slice-splits) for more usage examples. *Note:* - This evaluation set requires roughly 206 GBs of storage space. - It is possible to randomly access data by passing an index (e.g., `commfor_train[10]`, `commfor_train[247]`). - It may be wise to set `cache_dir` to some other directory if your home directory is limited. By default, it will download data to `~/.cache/huggingface/datasets`. - Not all images have a `prompt`. This can be because the generator does not require text prompts (e.g., unconditional, class-conditional) or due to an error. In cases where you need a specific portion of data, you can use the `.filter()` method (e.g., for data with prompts, `commfor_train.filter(lambda x: x['prompt'] != "", num_proc=8)`) # Dataset Creation ## Curation Rationale This dataset is created to address the limited model diversity of the existing datasets for generated image detection. While some existing datasets contain millions of images, they are typically sampled from handful of generator models. We instead sample 2.7M images from 4803 generator models, approximately 34 times more generators than the most extensive previous dataset that we are aware of. ## Collection Methodology We collect generators in three different subgroups. (1) We systematically download and sample open source latent diffusion models from Hugging Face. (2) We manually sample open source generators with various architectures and training procedures. (3) We sample from both open and closed commercially available generators. ## Personal and Sensitive Information The dataset does not contain any sensitive identifying information (i.e., does not contain data that reveals information such as racial or ethnic origin, sexual orientation, religious or political beliefs). # Considerations of Using the Data ## Social Impact of Dataset This dataset may be useful for researchers in developing and benchmarking forensics methods. Such methods may aid users in better understanding the given image. However, we believe the classifiers, at least the ones that we have trained or benchmarked, still show far too high error rates to be used directly in the wild, and can lead to unwanted consequences (e.g., falsely accusing an author of creating fake images or allowing generated content to be certified as real). ## Discussion of Biases The dataset has been primarily sampled from LAION captions. This may introduce biases that could be present in web-scale data (e.g., favoring human photos instead of other categories of photos). In addition, a vast majority of the generators we collect are derivatives of Stable Diffusion, which may introduce bias towards detecting certain types of generators. ## Other Known Limitations The generative models are sourced from the community and may contain inappropriate content. While in many contexts it is important to detect such images, these generated images may require further scrutiny before being used in other downstream applications. # Additional Information ## Acknowledgement We thank the creators of the many open source models that we used to collect the Community Forensics dataset. We thank Chenhao Zheng, Cameron Johnson, Matthias Kirchner, Daniel Geng, Ziyang Chen, Ayush Shrivastava, Yiming Dou, Chao Feng, Xuanchen Lu, Zihao Wei, Zixuan Pan, Inbum Park, Rohit Banerjee, and Ang Cao for the valuable discussions and feedback. This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0123. ## Licensing Information We release the dataset with a `cc-by-nc-sa-4.0` license for non-commercial research and educational purposes only. In addition, we note that each image in this dataset has been generated by the models with their respective licenses. We therefore provide metadata of all models present in our dataset with their license information. Please refer to the [metadata](https://huggingface.co/datasets/OwensLab/CommunityForensics/tree/main/data/metadata) for detailed licensing information for your specific application. We also attach licensing information for RAISE dataset included in this set [here](https://huggingface.co/datasets/OwensLab/CommunityForensics-Eval/blob/updating_v1/other/RAISE_License.pdf). <!-- Update this to be linking to the attached pdf file within the repository --> ## Citation Information Please cite our work as below if you used our dataset for your project. ``` @InProceedings{Park_2025_CVPR, author = {Park, Jeongsoo and Owens, Andrew}, title = {Community Forensics: Using Thousands of Generators to Train Fake Image Detectors}, booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)}, month = {June}, year = {2025}, pages = {8245-8257} } ``` Also, please cite the following real datasets which are included in this set. <details> <summary>Citation information for real datasets (Click)</summary> **RAISE** ```bibtex @inproceedings{dang2015raise, title={Raise: A raw images dataset for digital image forensics}, author={Dang-Nguyen, Duc-Tien and Pasquini, Cecilia and Conotter, Valentina and Boato, Giulia}, booktitle={Proceedings of the 6th ACM multimedia systems conference}, pages={219--224}, year={2015} } ``` **MS-COCO** ```bibtex @inproceedings{lin2014microsoft, title={Microsoft coco: Common objects in context}, author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence}, booktitle={European conference on computer vision}, pages={740--755}, year={2014}, organization={Springer} } ``` **FFHQ** ```bibtex @inproceedings{karras2019style, title={A style-based generator architecture for generative adversarial networks}, author={Karras, Tero and Laine, Samuli and Aila, Timo}, booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages={4401--4410}, year={2019} } ``` **LAION** ```bibtex @article{schuhmann2021laion, title={Laion-400m: Open dataset of clip-filtered 400 million image-text pairs}, author={Schuhmann, Christoph and Vencu, Richard and Beaumont, Romain and Kaczmarczyk, Robert and Mullis, Clayton and Katta, Aarush and Coombes, Theo and Jitsev, Jenia and Komatsuzaki, Aran}, journal={arXiv preprint arXiv:2111.02114}, year={2021} } ``` </details>The Community Forensics dataset aims to address the limited model diversity issue in existing datasets for generated image detection. The dataset collects 2.7 million images from 4803 generator models, approximately 34 times more generators than the most extensive previous dataset that we are aware of. It includes generated images paired with RAISE, COCO, FFHQ, and LAION datasets, and is used for training and evaluating fake image detection methods. The dataset is stored in the format of Parquet data frames, with each data instance containing image name, format, resolution, mode, image data, model name, NSFW flag, prompt, real source, subset, split, label, and architecture information. The dataset is released under the CC-BY-NC-SA-4.0 license for non-commercial research and educational purposes.

搜集汇总

数据集介绍

构建方式

在生成图像检测领域，现有数据集普遍受限于生成器模型的多样性不足。为突破这一瓶颈，OwensLab/CommunityForensics-Eval 数据集应运而生，其构建方式独具匠心。研究团队从三大维度系统收集生成器：首先，从 Hugging Face 平台大规模下载并采样开源潜在扩散模型；其次，手动收集涵盖多种架构与训练流程的开源生成器；最后，对商业可用的生成器进行采样，无论其是否开源。通过这一分层采集策略，数据集最终整合了 4803 个生成器模型，较以往最全面的数据集扩大了约 34 倍，并生成 270 万张图像，构建起一个规模宏大且模型多样性显著的评估基准。

使用方法

使用该数据集进行模型训练与评估时，需注意数据污染规避策略。建议采用数据集附带的 FFHQ 训练/测试划分文件，并仅使用 COCO 数据集的官方训练部分，RAISE 数据集则不应参与训练。在技术实现上，可通过 Hugging Face Datasets 库便捷加载：使用 `ds.load_dataset` 函数指定 `split='CompEval'` 即可获取评估集，图像数据以字节流形式存储，需借助 `PIL.Image.open` 与 `io.BytesIO` 进行解码。对于大规模应用，可启用 `streaming=True` 模式进行流式加载，并利用 `shuffle` 方法打乱数据顺序。此外，通过 `.filter()` 方法可筛选出含有提示文本的样本，满足特定实验需求。

背景与挑战

背景概述

随着生成式人工智能技术的迅猛发展，特别是扩散模型与生成对抗网络在图像合成领域的广泛应用，虚假图像检测已成为数字取证与信息安全研究中的核心议题。然而，现有基准数据集普遍受限于生成器模型的多样性不足——尽管部分数据集包含数百万张图像，却仅源自寥寥数种生成模型，导致检测方法在实际复杂场景下的泛化能力难以得到有效验证。为应对这一困境，OwensLab团队由Jeongsoo Park与Andrew Owens主导，于2025年在CVPR上发布了CommunityForensics-Eval数据集。该数据集从4803个生成模型中采样了约270万张图像，其生成器数量较此前最全面的数据集提升了约34倍，覆盖了从Hugging Face开源潜扩散模型到商业闭源系统的广泛架构，并系统性地配对了RAISE、COCO、FFHQ及LAION等真实图像源，旨在为虚假图像检测器提供更具挑战性与现实意义的评估基准。

当前挑战

CommunityForensics-Eval数据集所应对的领域挑战主要在于现有检测方法对生成器多样性的脆弱性——多数分类器在仅见过少数模型生成的图像时表现优异，但面对来自数千种不同架构、训练流程与商业系统的未知生成器时，检测精度急剧下降。该数据集的构建过程同样面临诸多困难：首先，需从海量社区模型中系统性地筛选并下载4803个生成器，确保其架构覆盖潜扩散模型、像素扩散模型、GAN及其他类型，同时处理各模型不同的许可证与使用限制；其次，在采样过程中需逐一生成图像并记录模型名称、提示词、NSFW标志等详尽元数据，以支撑可解释的评估；此外，为避免数据污染，需严格划分训练集与测试集，特别是针对FFHQ和COCO等常用数据集制定专用分割方案，并确保RAISE数据集不参与训练。这些挑战共同构成了当前虚假图像检测研究向真实世界部署迈进的关键瓶颈。

常用场景

经典使用场景

在深度伪造图像检测领域，CommunityForensics-Eval数据集作为一项里程碑式的基准评测资源，被广泛用于评估和比较不同伪造图像检测算法的泛化能力。该数据集汇聚了来自21种生成模型、涵盖RAISE、COCO、FFHQ和LAION等真实数据源配对的五万余张图像，为研究者提供了一个高度多样化且贴近真实场景的测试平台。其经典使用场景在于衡量检测器在面对未知生成模型时的鲁棒性，通过平均精度（mAP）和平均准确率（mAcc）等指标，系统性地揭示各类检测方法在跨模型、跨架构条件下的性能差异，从而推动伪造图像检测领域从单一模型验证向大规模、多源泛化评估的范式转变。

解决学术问题

该数据集精准回应了伪造图像检测研究中长期存在的模型多样性匮乏之困。既有数据集虽规模庞大，却往往仅依赖少数生成模型采样，导致检测器在应对新型或未见过生成器时性能骤降。CommunityForensics-Eval通过纳入4803个生成器模型——较此前最广泛数据集提升约34倍——并系统划分潜在扩散、像素扩散、生成对抗网络及商业模型等架构类别，为学术研究提供了验证检测方法跨架构泛化性的关键工具。它解决了如何设计不依赖于特定生成痕迹、具有真实世界实用价值的通用检测器这一核心问题，其发布深刻影响了数字取证领域对模型泛化能力的认知标准，并推动了更为稳健的检测范式构建。

实际应用

在实际应用层面，该数据集为内容审核平台、新闻媒体机构及司法取证系统提供了不可或缺的检测基准。随着生成式AI技术的泛滥，虚假图像可能被用于传播不实信息、伪造证据或实施欺诈。CommunityForensics-Eval所模拟的复杂生成源环境，使得基于其训练的检测器能够更有效地在社交媒体、新闻报道及法律证据等场景中甄别合成内容。尽管当前检测器在开放世界中的误差率仍不容忽视，该数据集通过暴露检测方法在面对商业级黑盒模型（如Midjourney、DALL·E）时的真实表现，为安全系统开发者提供了优化阈值、降低误报率的实验依据，从而间接增强了社会对数字内容真实性的甄别能力。

数据集最近研究

最新研究方向

在深度伪造图像检测领域，现有数据集往往局限于少数生成模型，难以应对生成技术快速迭代带来的泛化挑战。OwensLab/CommunityForensics-Eval数据集应运而生，其核心创新在于从超过4800个生成模型中采样了270万张图像，覆盖了潜在扩散模型、生成对抗网络及商业闭源系统等多种架构，极大提升了检测器的训练多样性。该数据集已被CVPR 2025录用，其配套的检测方法在综合评估集上取得了0.987的平均精度，显著优于此前基准。这一工作不仅推动了伪造图像检测从封闭场景向开放世界的跨越，也为应对生成式AI滥用引发的虚假信息泛滥提供了关键验证平台，具有重要的社会安全意义。