ArGuard-Task1

# ArGuard – Track A: Arabic Hateful Memes This repository hosts the official dataset for **Track A** of the **ArGuard** shared task: multimodal hateful-meme detection in Arabic. Each instance is an Arabic meme (image + OCR-extracted overlaid text) manually annotated for hatefulness and fine-grained sub-types. > **Content warning.** The dataset contains text and imagery that is > offensive, discriminatory, or otherwise harmful by design. Handle > with care. ## Track A subtasks Given a meme (image + Arabic text): - **Subtask A1 – Binary.** Classify the meme as `Hateful` or `Not Hateful`. - **Subtask A2 – Fine-grained category prediction (multi-label).** Predict the applicable fine-grained sub-type(s) from a unified taxonomy that covers both hateful and non-hateful categories. Hateful memes draw labels from the hateful sub-type set (Mocking, Incitement, Dehumanization, Slurs, Contempt, Inferiority, Exclusion, …); non-hateful memes draw from `Humor`, `Sarcasm`, plus the shared `Other`. Both subtasks are evaluated on every meme. ## Splits | Split | Records | Labels | Source | Released | |--------------|---------|--------------|-----------------------------------------|---------------------------------| | `train` | 3,500 | full | single-annotated bulk | development phase | | `dev` | 500 | full | single-annotated bulk | development phase | | `dev_test` | 500 | full | single-annotated test sample | development phase (leaderboard); **labels released** | | `test` | 500 | full | **triple-annotated gold** (calibration) | final-evaluation phase; **labels released** | - `dev_test` was the **leaderboard set** for the development phase. That phase is now closed and its gold labels have been **released**, so `dev_test` is usable as additional labelled data. It is disjoint from `test`. - `test` was the **blind test set** for the final-evaluation phase. That phase has now closed and the gold labels are **released**, so the full dataset is labelled and the benchmark can be evaluated offline. All 500 records are triple-annotated with majority voting, which makes `test` the highest-quality split and the one to report results on. ### Binary label distribution | Split | Hateful | Not Hateful | % Hateful | |------------|--------:|------------:|----------:| | train | 1,324 | 2,176 | 37.8% | | dev | 189 | 311 | 37.8% | | dev_test | 189 | 311 | 37.8% | | test | 148 | 352 | 29.6% | | **Total** | **1,850** | **3,150** | 37.0% | Note that `test` is deliberately **not** distribution-matched to the other splits: it was drawn from the triple-annotated calibration pool, where majority voting resolves more borderline cases to `Not Hateful`. ### Fine-grained sub-types (Subtask A2) The Subtask A2 label space is **one unified multi-label vocabulary** that covers both hateful and non-hateful sub-types: - **Hateful sub-types** (active in the released data): Mocking, Incitement, Dehumanization, Slurs, Contempt, Inferiority, Exclusion. - **Non-hateful sub-types**: Humor, Sarcasm. - **Shared**: Other (used by both Hateful and Not-Hateful memes). Five additional hateful classes appear in the annotation taxonomy but have **zero training support** in the released data: Extremism, Historical, Insults, Stereotyping, Threat. They are documented for completeness, accepted by the format checker, and ignored by the scorer. In practice each meme's fine-grained labels are drawn from its own binary class: a Hateful meme will only carry hateful sub-types (and/or `Other`); a Not-Hateful meme will only carry `Humor` / `Sarcasm` / `Other`. Sub-types are multi-label, so per-class counts sum to more than the meme counts. ## Record schema ```python { "id": "f9a8…b1.jpg", # str – original image filename, unique "image": <PIL.Image.Image>, # embedded bytes, decoded on access "text": "…", # str – OCR-extracted Arabic meme text "label": "Hateful" | "Not Hateful", # every split is now labelled "fine_grained_label": [...], # list[str], possibly empty } ``` ## Usage ```python from datasets import load_dataset ds = load_dataset("QCRI/ArGuard-Task1") print(ds) train_ex = ds["train"][0] train_ex["image"].show() print(train_ex["text"], train_ex["label"], train_ex["fine_grained_label"]) test_ex = ds["test"][0] print(test_ex["label"], test_ex["fine_grained_label"]) ``` **Recommended benchmark protocol.** Train on `train` + `dev` + `dev_test` (4,500 memes, single-annotator labels) and report on `test` (500 memes, triple-annotated majority vote). That is the split the shared-task leaderboard was scored on, so results are comparable to the published rankings. ## Shared-task resources - **Website:** https://araieval.github.io/ArGuard2026/ - **Starter kit / baselines / scorers (GitHub):** https://github.com/araieval/ArGuard-2026-tasks - **Submission — Subtask A1:** https://www.codabench.org/competitions/16909/ - **Submission — Subtask A2:** https://www.codabench.org/competitions/16910/ - **Contact organisers:** [e-mail] ## Annotation - All memes are manually annotated following the ArGuard guidelines. - **train**, **dev**, **dev_test**: single-annotator labels (bulk annotation). - **test**: triple-annotated. Binary label is the majority vote; the fine-grained label set is the union of sub-types selected by annotators whose binary label matches the majority. - Inter-annotator agreement on the calibration subset is above 0.81. ## Intended use and limitations - **Intended use.** Research on Arabic multimodal hate speech detection, including binary classification, fine-grained sub-type prediction, and vision-language modelling. - **Limitations.** Memes reflect online discourse and contain offensive and harmful content. Annotations on `train` / `dev` / `dev_test` are single-annotator and may contain noise; only the held-out `test` split uses triple-annotated majority-voted labels. - **Not for deployment.** This dataset is for research and benchmarking; it is not a moderation tool. ## License Released under **CC BY-NC 4.0** for non-commercial research use only. Not to be used for commercial purposes or for training systems that generate harmful content. ## Citation A citation will be provided when the shared-task overview paper is released. Until then, please cite this repository URL. ## Contact - **Email:** [e-mail] - **Website:** https://araieval.github.io/ArGuard2026/ - **GitHub:** https://github.com/araieval/ArGuard-2026-tasks