MemeLens

# MemeLens A large-scale multilingual multimodal meme understanding benchmark with 46 classification tasks across 9 languages, enriched with LLM-generated explanations and LLM-as-Judge quality scores. This is the VLM (Vision-Language Model) version of [MemeLens](https://huggingface.co/datasets/QCRI/MemeLens), extended with natural language explanations for each sample and automated quality evaluation via LLM-as-Judge. - **Paper:** [MemeLens: Multilingual Multitask VLMs for Memes](https://huggingface.co/papers/2601.12539) - **Code:** [MohamedBayan/MemeLens](https://github.com/MohamedBayan/MemeLens) ## Dataset Overview | Statistic | Value | |-----------|-------| | Total samples | 271,835 | | Datasets/Tasks | 46 | | Languages | 9 (ar, bn, de, en, es, hi, ro, ru, zh) | | Splits | train / test / val | ## Quick Start ### Download the Dataset Use the following commands from the official repository to download the dataset: ```bash # Download everything (~88GB with images) python data/download_dataset.py # Download specific language(s) python data/download_dataset.py --languages en ar # Download specific dataset(s) python data/download_dataset.py --datasets Hateful_en_FHM abuse_bn__BanglaAbuseMeme # List available datasets python data/download_dataset.py --list ``` ## Structure The dataset is organized by language: ``` {language}/ {dataset_name}/ images/ train.jsonl test.jsonl val.jsonl ``` ## Fields **All splits:** | Field | Description | |-------|-------------| | `id` | Unique sample identifier | | `image` | Relative path to the meme image | | `text` | OCR/extracted text from the meme | | `label` | Classification label for the task | | `task_description` | English description of the classification task | | `explanation` | LLM-generated English explanation justifying the label | | `native_label` | (multilingual only) Label in the meme's native language | | `native_task_description` | (multilingual only) Task description in native language | | `native_explanation` | (multilingual only) Explanation in native language | **Test split only (LLM-as-Judge):** | Field | Description | |-------|-------------| | `informativeness` | Average judge score (1–5) from GPT-5 and Gemini-2.5-Pro | | `clarity` | Average judge score (1–5) from GPT-5 and Gemini-2.5-Pro | | `plausibility` | Average judge score (1–5) from GPT-5 and Gemini-2.5-Pro | | `faithfulness` | Average judge score (1–5) from GPT-5 and Gemini-2.5-Pro | | `llm_judge` | Per-criterion scores and justifications from each judge model | ## Languages and Tasks | Language | # Tasks | Datasets | |----------|---------|----------| | Arabic (ar) | 2 | Hateful_ar__Prop2Hate-Meme, propoganda_ar_ArMeme | | Bengali (bn) | 5 | abuse, sarcasm, sentiment, vulgar (BanglaAbuseMeme), Hateful (MUTE) | | German (de) | 1 | Hateful_de__Multi3Hate | | English (en) | 23 | HarMeme, FHM, MMHS, MAMI, memotion, MET_Meme, Multi3Hate, MIMIC | | Spanish (es) | 1 | Hateful_es__Multi3Hate | | Hindi (hi) | 3 | Hateful (Multi3Hate), Misogyny, Misogyny_Categories (MIMIC2024) | | Romanian (ro) | 4 | deepfake, emotion, political, sentiment (RoMemes) | | Russian (ru) | 1 | toxic_ru__Toxic_Memes_Detection_Dataset | | Chinese (zh) | 6 | Hateful (Multi3Hate), intention, metaphor, offensiveness, sentiment (MET_Meme) | ## Citation ```bibtex @inproceedings{shahroor-etal-2026-memelens, title = {{MemeLens}: Multilingual Multitask {VLM}s for Memes}, author = {Shahroor, Ali Ezzat and Kmainasi, Mohamed Bayan and Hasnat, Abul and Dimitrov, Dimitar and Da San Martino, Giovanni and Nakov, Preslav and Alam, Firoj}, url={ttps://arxiv.org/abs/2601.12539} } ``` ## Related - **Dataset (classification only):** [QCRI/MemeLens](https://huggingface.co/datasets/QCRI/MemeLens) - **Paper:** [arXiv:2601.12539](https://arxiv.org/abs/2601.12539)