laraibnoorien/SAMM

---
license: apache-2.0
task_categories:
- token-classification
- text-classification
- image-text-to-text
- object-detection
language:
- en
size_categories:
- 100K<n<1M
tags:
- Multimodal
- manipulation-detection
- media-forensics
- deepfake-detection
---

# Resource 📖

<h4 align="center">
  <a href="https://arxiv.org/abs/2509.12653">[ACM MM Paper]</a> | <a href="https://huggingface.co/datasets/SJJ0854/SAMM">[SAMM HF]</a> | <a href="https://huggingface.co/datasets/SJJ0854/CAP">[CAP HF]</a> | <a href=" https://github.com/shen8424/SAMM-RamDG-CAP">[Github Code]</a>
</h4>

# Notes ⚠️

- If you want to import the CAP data into your own dataset, please refer to [this](https://github.com/shen8424/CAP).
- If you want to run RamDG on datasets other than SAMM and use CNCL to incorporate external knowledge, please ensure to configure ```idx_cap_texts``` and ```idx_cap_images``` in the dataset jsons.
- We have upgraded the SAMM JSON files. The latest versions (SAMM with CAP or without CAP) are available on July 24, 2025. Please download the newest version.

# Brief introduction

<div align="center">
<img src='./figures/teaser.png' width='90%'>
</div>

We present <b>SAMM</b>, a large-scale dataset for Detecting and Grounding Semantic-Coordinated Multimodal Manipulation. The official code has been released at [this](https://github.com/shen8424/SAMM-RamDG-CAP).

**Dataset Statistics:**

<div align="center">
<img src='./figures/samm_statistics.png' width='90%'>
</div>

# Annotations
```
{
"text": "Lachrymose Terri Butler, whose letter prompted Peter Dutton to cancel Troy Newman's visa, was clearly upset.",
"fake_cls": "attribute_manipulation",
"image": "emotion_jpg/65039.jpg",
"id": 13,
"fake_image_box": [
665,
249,
999,
671
],
"cap_texts": {
"Terri Butler": "Terri Butler Gender: Female, Occupation: Politician, Birth year: 1977, Main achievement: Member of Australian Parliament.",
"Peter Dutton": "Peter Dutton Gender: Male, Occupation: Politician, Birth year: 1970, Main achievement: Australian Minister for Defence."
},
"cap_images": {
"Terri Butler": "Terri Butler",
"Peter Dutton": "Peter Dutton"
},
"idx_cap_texts": [
1,
0
],
"idx_cap_images": [
1,
0
],
"fake_text_pos": [
0,
11,
13,
14,
15
]
}
```

- `image`: The relative path to the original or manipulated image.
- `text`: The original or manipulated text caption.
- `fake_cls`: Indicates the type of manipulation (e.g., forgery, editing).
- `fake_image_box`: The bounding box coordinates of the manipulated region in the image.
- `fake_text_pos`: A list of indices specifying the positions of manipulated tokens within the `text` string.
- `cap_texts`: Textual information extracted from CAP (Contextual Auxiliary Prompt) annotations.
- `cap_images`: Relative paths to visual information from CAP annotations.
- `idx_cap_texts`: A binary array where the i-th element indicates whether the i-th celebrity in `cap_texts` is tampered (1 = tampered, 0 = not tampered).
- `idx_cap_images`: A binary array where the i-th element indicates whether the i-th celebrity in `cap_images` is tampered (1 = tampered, 0 = not tampered).

# 🤗🤗🤗 Citation
```
@misc{shen2025artificialmisalignmentdetectinggrounding,
title={Beyond Artificial Misalignment: Detecting and Grounding Semantic-Coordinated Multimodal Manipulations},
author={Jinjie Shen and Yaxiong Wang and Lechao Cheng and Nan Pu and Zhun Zhong},
year={2025},
eprint={2509.12653},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2509.12653},
}
```