vs16/counter-hate-dataset

--- dataset_info: features: - name: original_sample_id dtype: string - name: counterfactual_id dtype: string - name: text dtype: string - name: class_label dtype: string - name: target_group dtype: string - name: polarity dtype: string - name: hate_score dtype: float32 - name: confidence dtype: float32 - name: cf_type dtype: string - name: t2i_prompt dtype: string - name: image_path dtype: string - name: split dtype: string splits: - name: train num_bytes: 4624365 num_examples: 12597 - name: validation num_bytes: 1007932 num_examples: 2700 - name: test num_bytes: 1010867 num_examples: 2703 license: cc-by-4.0 size_gb: 2.3 --- # Counter-Hate Dataset A large-scale multimodal dataset for studying fairness and bias in hate speech detection systems with counterfactual augmentation. **🎯 Target venue:** ACM Multimedia 2026 ## Dataset Description This dataset contains **18,000 text-image pairs** categorized into 8 hate speech classes with varying levels of protected group representation. The dataset was created to evaluate whether Counterfactual Data Augmentation (CDA) introduces or amplifies bias in hate speech detection models. ### Key Features: - **Multimodal**: Text + Generated Images (from T2I models like Z-Image-Turbo) - **18,000 samples**: 6,000 original texts + 12,000 counterfactual variants via identity-term substitution - **8 hate speech classes**: - hate_race, hate_religion, hate_gender, hate_other - offensive_non_hate, neutral_discussion, counter_speech, ambiguous - **8 protected groups**: - race/ethnicity, religion, gender, sexual_orientation, national_origin/citizenship, disability, age, multiple/none - **Stratified splits: train/val/test** (70% / 15% / 15%) with originals only in val/test - **Experimental conditions**: nCF (6K originals) and CF (18K with counterfactuals) ## Dataset Structure ``` counter-hate-dataset/ ├── data/ │ ├── train.csv # 12,597 training samples (includes counterfactuals) │ ├── validation.csv # 2,700 validation samples (originals only) │ ├── test.csv # 2,703 test samples (originals only) │ └── dataset.csv # Combined all samples ├── images/ │ ├── hate/ # 9,000 images for hate speech samples │ └── non_hate/ # 9,000 images for non-hate samples └── README.md # This file ``` ## Features Each row in the CSV files contains: | Column | Type | Description | |--------|------|-------------| | `original_sample_id` | str | Unique identifier for the original sample (e.g., HS_HATE_RACE_0001) | | `counterfactual_id` | str | Unique ID for this variant (original or counterfactual_X) | | `text` | str | The actual text content | | `class_label` | str | One of 8 hate speech categories | | `target_group` | str | Protected group mentioned in text (8 groups) | | `polarity` | str | 'hate', 'non-hate' | | `hate_score` | float | Numeric hate score from annotations | | `confidence` | float | Confidence in the annotation (0-10 scale) | | `cf_type` | str | 'original' or 'counterfactual_1', 'counterfactual_2', etc. | | `t2i_prompt` | str | Text-to-image generation prompt | | `image_path` | str | Relative path to corresponding PNG image | | `split` | str | 'train', 'validation', or 'test' | ## Splits Training conditions and split statistics: ### Condition nCF (No Counterfactual) - Train: 4,158 original samples - Validation: 891 original samples - Test: 892 original samples - **Total: 5,841 samples** ### Condition CF (With Counterfactual) - Train: 12,597 samples (includes counterfactuals) - Validation: 2,700 samples (originals only) - Test: 2,703 samples (originals only) - **Total: 18,000 samples** All splits are stratified by `class_label` to preserve class distribution. ## Class Distribution Each class contains 750 original samples: - **Hate Speech (4 classes)**: - hate_race: 750 - hate_religion: 750 - hate_gender: 750 - hate_other: 750 - **Non-Hate (4 classes)**: - offensive_non_hate: 750 - neutral_discussion: 750 - counter_speech: 750 - ambiguous: 750 ## Image Generation Images were generated using the **Z-Image-Turbo** text-to-image model with prompts derived from the text content. Each image corresponds to a text sample and represents the hate speech category and context. Images are stored as PNG files organized by category. ### Image Organization ``` images/ ├── hate/ │ ├── Hate_Gender/ # Gender-targeted hate speech │ ├── Hate_Others/ # Other hate categories │ ├── Hate_race/ # Race-targeted hate speech │ └── ... └── non_hate/ ├── generated_images-ambigious/ ├── generated_images-counter-speech/ ├── generated_images-neutral/ └── ... ``` ## Usage ### Load with Hugging Face Datasets ```python from datasets import load_dataset # Load the full dataset dataset = load_dataset('vs16/counter-hate-dataset') # Access specific split train_data = dataset['train'] val_data = dataset['validation'] test_data = dataset['test'] # Access a sample sample = train_data[0] print(sample['text']) print(sample['image_path']) ``` ### Load with Pandas ```python import pandas as pd from PIL import Image # Load a specific split train_df = pd.read_csv('data/train.csv') # Access a row sample = train_df.iloc[0] print(sample['text']) print(sample['class_label']) # Load the image image = Image.open(f"images/{sample['image_path']}") image.show() ``` ### Stratified Train/Val/Test Split ```python import pandas as pd from sklearn.model_selection import train_test_split df = pd.read_csv('data/dataset.csv') # Using the provided split column train_df = df[df['split'] == 'train'] val_df = df[df['split'] == 'validation'] test_df = df[df['split'] == 'test'] ``` ## Experimental Methodology ### Counterfactual Generation Counterfactual samples were generated via **identity-term substitution**: 1. Extract identity terms (group descriptors) from original text 2. Replace with alternative terms for the same attribute dimension 3. Examples: - "Muslim" → "Christian", "Jewish", "Hindu", etc. - "Black" → "Asian", "Hispanic", "Native American", etc. - "woman" → "man", etc. ### Data Preparation (Conditions) - **nCF (No Counterfactual)**: Only original 6,000 samples - **CF (With Counterfactual)**: 6,000 originals + 12,000 counterfactual variants ### Fair Evaluation Protocol - Validation and test sets contain **only original samples** for both conditions - Training set includes counterfactuals in CF condition - Prevents data leakage and ensures comparable evaluation ## Citation If you use this dataset in your research, please cite: ```bibtex @dataset{vs16_counter_hate_2026, title={Counter-Hate Dataset: A Multimodal Benchmark for Studying Fairness and Bias in Hate Speech Detection}, year={2026}, publisher={Hugging Face Datasets}, url={https://huggingface.co/datasets/vs16/counter-hate-dataset} } ``` And the original work: ```bibtex @article{kennedy2020measuring, title={Measuring the Reliability of Hate Speech Annotations: The Case of the European Parliament Debates}, author={Kennedy, Bing and Atkinson, David and others}, year={2020} } ``` ## License The dataset is provided for **research purposes only**. Users must comply with applicable laws and ethical guidelines when working with this data. ## Ethical Considerations ⚠️ **Important Note**: This dataset contains hate speech and offensive language for research purposes only. ### Responsible Use Guidelines 1. **Research Purpose Only**: Use solely for studying bias, fairness, and detection systems 2. **Do Not Amplify**: Do not use to train systems that amplify or spread hate speech 3. **Sensitivity**: Be aware of the sensitive nature of the content 4. **Attribution**: Always cite and credit the dataset source 5. **Report Issues**: Report any misuse or ethical concerns 6. **Institutional Review**: Consider IRB approval for related human studies ### Protected Groups Representation The dataset explicitly includes diverse protected groups to ensure comprehensive bias evaluation. This is intentional and necessary for fairness research. ## Dataset Statistics - **Total samples**: 18,000 text-image pairs - **Original samples**: 6,000 - **Counterfactual variants**: 12,000 - **Image files**: 18,000 PNG images - **Total size**: ~2.3 GB (with images) - **CSV size**: ~8 MB (combined) - **Average text length**: 150-300 characters - **Image resolution**: 512x512px (typical for T2I models) ## Reproducibility All splits and train/val/test assignments are deterministic and reproducible: - **Random seed**: 42 - **Stratification**: By class_label (8 classes) - **Split level**: By original_sample_id (group-level splitting) - **Canonical source**: `canonical_splits.json` defines all splits ## Known Limitations 1. Images are synthetically generated and may not perfectly represent real-world scenarios 2. Counterfactual generation via term substitution is limited to identity/demographic attributes 3. Limited to English text 4. Hate speech prevalence is intentionally high (for detection research) 5. May not generalize to all hate speech types or contexts ## Questions & Support For issues, questions, or suggestions: - 🐛 Dataset issues: Report on GitHub - 📧 Contact: via Hugging Face dataset page - 💬 Discussions: Use the Hugging Face Discussions tab ## Related Work This dataset builds on and relates to: - UCBerkeley-DLab's "Measuring Hate Speech" dataset - HateBERT and other hate speech detection models - Fairness and bias research in NLP - Counterfactual data augmentation literature --- **Dataset Version**: 1.0 **Last Updated**: April 2, 2026 **Status**: Ready for research use数据集信息： 特征： - 名称：original_sample_id，数据类型：字符串（string） - 名称：counterfactual_id，数据类型：字符串 - 名称：text，数据类型：字符串 - 名称：class_label，数据类型：字符串 - 名称：target_group，数据类型：字符串 - 名称：polarity，数据类型：字符串 - 名称：hate_score，数据类型：单精度浮点数（float32） - 名称：confidence，数据类型：单精度浮点数 - 名称：cf_type，数据类型：字符串 - 名称：t2i_prompt，数据类型：字符串 - 名称：image_path，数据类型：字符串 - 名称：split，数据类型：字符串 划分集： - 名称：训练集（train），字节数：4624365，样本数：12597 - 名称：验证集（validation），字节数：1007932，样本数：2700 - 名称：测试集（test），字节数：1010867，样本数：2703 许可协议：知识共享署名4.0（cc-by-4.0） 总大小：2.3GB --- # 反仇恨数据集（Counter-Hate Dataset） 用于借助反事实数据增强（Counterfactual Data Augmentation, CDA）研究仇恨言论检测系统公平性与偏倚的大规模多模态数据集。 **🎯 投稿目标会议**：ACM Multimedia 2026 ## 数据集概述 本数据集包含**18000个文本-图像对**，分为8个仇恨言论类别，覆盖不同程度的受保护群体表征。本数据集旨在评估反事实数据增强（Counterfactual Data Augmentation, CDA）是否会在仇恨言论检测模型中引入或放大偏倚。 ### 核心特性： - **多模态属性**：文本+由文本到图像（Text-to-Image, T2I）模型（如Z-Image-Turbo）生成的图像 - **18000个样本**：6000个原始文本 + 12000个通过身份术语替换生成的反事实变体样本 - **8类仇恨言论类别**： - 种族仇恨（hate_race）、宗教仇恨（hate_religion）、性别仇恨（hate_gender）、其他仇恨（hate_other） - 冒犯性非仇恨内容（offensive_non_hate）、中性讨论（neutral_discussion）、反仇恨言论（counter_speech）、模糊类别（ambiguous） - **8类受保护群体**： - 种族/民族、宗教、性别、性取向、国籍/公民身份、残疾、年龄、多群体/无群体 - **分层划分集：训练/验证/测试**（70% / 15% / 15%），验证集与测试集仅包含原始样本 - **两种实验条件**：无反事实样本组（nCF，6000个原始样本）与含反事实样本组（CF，18000个含反事实样本的全集） ## 数据集结构 counter-hate-dataset/ ├── data/ │ ├── train.csv # 12597条训练样本（包含反事实样本） │ ├── validation.csv # 2700条验证样本（仅原始样本） │ ├── test.csv # 2703条测试样本（仅原始样本） │ └── dataset.csv # 所有样本的合并文件 ├── images/ │ ├── hate/ # 9000张仇恨言论样本对应图像 │ └── non_hate/ # 9000张非仇恨言论样本对应图像 └── README.md # 本说明文件 ## 字段说明 CSV文件中的每一行包含以下字段： | 字段名 | 数据类型 | 描述 | |--------|------|-------------| | `original_sample_id` | 字符串 | 原始样本的唯一标识符（例如：HS_HATE_RACE_0001） | | `counterfactual_id` | 字符串 | 该样本变体的唯一ID（原始样本或counterfactual_X格式） | | `text` | 字符串 | 实际文本内容 | | `class_label` | 字符串 | 8类仇恨言论类别之一 | | `target_group` | 字符串 | 文本中提及的受保护群体（共8类） | | `polarity` | 字符串 | 标注极性，可选'hate'（仇恨）或'non-hate'（非仇恨） | | `hate_score` | 浮点数 | 标注人员给出的仇恨程度评分 | | `confidence` | 浮点数 | 标注置信度（0-10的评分范围） | | `cf_type` | 字符串 | 样本类型，可选'original'（原始样本）或'counterfactual_1'、'counterfactual_2'等 | | `t2i_prompt` | 字符串 | 文本到图像生成的提示词 | | `image_path` | 字符串 | 对应PNG图像的相对路径 | | `split` | 字符串 | 样本所属划分集，可选'train'、'validation'或'test' | ## 划分集设置 训练条件与划分统计信息： ### 无反事实样本组（nCF, No Counterfactual） - 训练集：4158条原始样本 - 验证集：891条原始样本 - 测试集：892条原始样本 - **总样本数：5841条** ### 含反事实样本组（CF, With Counterfactual） - 训练集：12597条样本（包含反事实样本） - 验证集：2700条样本（仅原始样本） - 测试集：2703条样本（仅原始样本） - **总样本数：18000条** 所有划分集均按`class_label`进行分层采样，以保留类别分布。 ## 类别分布 每个类别包含750条原始样本： - **仇恨言论类别（4类）**： - hate_race（种族仇恨）：750条 - hate_religion（宗教仇恨）：750条 - hate_gender（性别仇恨）：750条 - hate_other（其他仇恨）：750条 - **非仇恨言论类别（4类）**： - offensive_non_hate（冒犯性非仇恨内容）：750条 - neutral_discussion（中性讨论）：750条 - counter_speech（反仇恨言论）：750条 - ambiguous（模糊类别）：750条 ## 图像生成 图像通过**Z-Image-Turbo**文本到图像模型生成，提示词源自文本内容。每张图像对应一条文本样本，还原其仇恨言论类别与上下文场景。图像按类别组织存储为PNG格式文件。 ### 图像组织方式 images/ ├── hate/ │ ├── Hate_Gender/ # 针对性别的仇恨言论图像 │ ├── Hate_Others/ # 其他仇恨类别的图像 │ ├── Hate_race/ # 针对种族的仇恨言论图像 │ └── ... └── non_hate/ ├── generated_images-ambigious/ ├── generated_images-counter-speech/ ├── generated_images-neutral/ └── ... ## 使用方法 ### 通过Hugging Face Datasets加载 python from datasets import load_dataset # 加载完整数据集 dataset = load_dataset('vs16/counter-hate-dataset') # 访问指定划分集 train_data = dataset['train'] val_data = dataset['validation'] test_data = dataset['test'] # 访问单条样本 sample = train_data[0] print(sample['text']) print(sample['image_path']) ### 通过Pandas加载 python import pandas as pd from PIL import Image # 加载指定划分集 train_df = pd.read_csv('data/train.csv') # 访问单条样本 sample = train_df.iloc[0] print(sample['text']) print(sample['class_label']) # 加载对应图像 image = Image.open(f"images/{sample['image_path']}") image.show() ### 分层训练/验证/测试划分 python import pandas as pd from sklearn.model_selection import train_test_split df = pd.read_csv('data/dataset.csv') # 使用数据集中自带的split字段进行划分 train_df = df[df['split'] == 'train'] val_df = df[df['split'] == 'validation'] test_df = df[df['split'] == 'test'] ## 实验方法 ### 反事实样本生成 反事实样本通过**身份术语替换**生成： 1. 从原始文本中提取身份术语（群体描述词） 2. 将其替换为同一属性维度下的替代术语 3. 示例： - ‘穆斯林’ → ‘基督徒’、‘犹太教徒’、‘印度教徒’等 - ‘黑人’ → ‘亚裔’、‘西班牙裔’、‘北美原住民’等 - ‘女性’ → ‘男性’等 ### 数据准备（实验条件） - **nCF（无反事实样本）**：仅使用6000条原始样本 - **CF（含反事实样本）**：6000条原始样本 + 12000条反事实变体样本 ### 公平评估协议 - 验证集与测试集在两种实验条件下均仅包含原始样本 - 训练集在CF条件下包含反事实样本 - 该设计可避免数据泄露，确保评估结果具有可比性 ## 引用说明 若您在研究中使用本数据集，请引用： bibtex @dataset{vs16_counter_hate_2026, title={Counter-Hate Dataset: A Multimodal Benchmark for Studying Fairness and Bias in Hate Speech Detection}, year={2026}, publisher={Hugging Face Datasets}, url={https://huggingface.co/datasets/vs16/counter-hate-dataset} } 以及原始研究文献： bibtex @article{kennedy2020measuring, title={Measuring the Reliability of Hate Speech Annotations: The Case of the European Parliament Debates}, author={Kennedy, Bing and Atkinson, David and others}, year={2020} } ## 许可协议 本数据集仅用于**学术研究目的**。使用者在处理本数据时需遵守相关法律法规与伦理准则。 ## 伦理考量 ⚠️ **重要提示**：本数据集包含仇恨言论与冒犯性内容，仅用于学术研究。 ### 负责任使用准则 1. **仅限研究用途**：仅用于研究偏倚、公平性与检测系统 2. **禁止放大仇恨**：不得用于训练会放大或传播仇恨言论的系统 3. **敏感内容注意**：注意内容的敏感性 4. **署名要求**：始终引用并标注数据集来源 5. **问题反馈**：报告任何误用或伦理问题 6. **伦理审查**：相关人类受试者研究需考虑伦理审查委员会（IRB）审批 ### 受保护群体表征 本数据集明确包含多样化的受保护群体，以确保全面的偏倚评估。该设计为刻意为之，是公平性研究的必要设置。 ## 数据集统计信息 - **总样本数**：18000个文本-图像对 - **原始样本数**：6000条 - **反事实变体样本数**：12000条 - **图像文件数**：18000张PNG图像 - **总大小**：约2.3GB（包含图像） - **CSV文件总大小**：约8MB（合并后） - **平均文本长度**：150-300个字符 - **图像分辨率**：512×512像素（文本到图像模型的典型输出分辨率） ## 可复现性说明 所有划分集与训练/验证/测试分配均为确定性设置，可复现： - **随机种子**：42 - **分层依据**：按class_label（8个类别）分层 - **划分粒度**：按original_sample_id进行分组划分 - **官方划分文件**：`canonical_splits.json`中定义了所有划分集 ## 已知局限性 1. 图像为合成生成，可能无法完全还原真实场景 2. 通过术语替换生成的反事实样本仅局限于身份/人口统计属性 3. 仅支持英文文本 4. 仇恨言论占比刻意设置较高（以适配检测研究需求） 5. 可能无法推广至所有仇恨言论类型或场景 ## 问题与支持 若您遇到问题、有疑问或建议： - 🐛 数据集相关问题：在GitHub提交报告 - 📧 联系方式：通过Hugging Face数据集页面联系 - 💬 讨论：使用Hugging Face讨论板块 ## 相关研究 本数据集基于并关联以下研究： - 加州大学伯克利分校D实验室的「Measuring Hate Speech」数据集 - HateBERT及其他仇恨言论检测模型 - 自然语言处理领域的公平性与偏倚研究 - 反事实数据增强相关文献 --- **数据集版本**：1.0 **最后更新时间**：2026年4月2日 **状态**：可用于学术研究