AynVQA

# Ayn-VQA-ArabicNLP26 👁️ **Ayn-VQA-ArabicNLP26** is a culturally grounded Arabic multimodal evaluation dataset, part of the [ImageEval 2026 Shared Task](https://imageeval2026.github.io/) at ArabicNLP 2026. Ayn (عين, "eye") tests whether a model can read a culturally specific image, both from a spoken Arabic question and by telling grounded descriptions apart from plausible but hallucinated ones. Each task is offered as two language tracks, **English** and **Modern Standard Arabic (MSA)**, scored separately. ## 💬 Join our Slack Join the **[ImageEval Slack channel](https://join.slack.com/t/mm-eval/shared_invite/zt-44zii4e53-IiZZ3O73Za9XL4umb9mjUQ)** for announcements about data releases, deadlines, and updates, and to connect with the organisers and other participants. ## 🎯 Tasks **Spoken VQA.** Given an image and the spoken question and options (audio), choose the correct option. Prediction: the option index 0, 1 or 2. **Hallucination detection.** Given an image and three statements, decide for **each** statement whether it is **True** (grounded in the image) or **False** (a hallucination). Exactly one statement is grounded. Prediction: a True/False label per statement. ## 🗂️ Subsets | config | task | language | Codabench | |---|---|---|---| | `task1a_en` | Spoken VQA | English | [compete](https://www.codabench.org/competitions/17049/) | | `task1a_msa` | Spoken VQA | MSA | [compete](https://www.codabench.org/competitions/17048/) | | `task1b_en` | Hallucination | English | [compete](https://www.codabench.org/competitions/17051/) | | `task1b_msa` | Hallucination | MSA | [compete](https://www.codabench.org/competitions/17050/) | The English and MSA tracks of a task are parallel: same images, same answers, and the questions are translations of each other. ## 🌍 Countries The dataset spans **18 Arab countries**: Algeria, Bahrain, Egypt, Iraq, Jordan, Kuwait, Lebanon, Libya, Morocco, Oman, Palestine, Qatar, Saudi Arabia, Sudan, Syria, Tunisia, UAE, Yemen. ## 🔊 Audio The Task 1a questions in `train`, `dev` and `devtest` are **synthetically generated** using voice cloning (TTS). The questions in the final blind **test** set will be **human-recorded**; expect a speaker/recording-condition shift between the dev-phase audio and the test audio. ## 📂 Files ``` images/<id>.jpg one image per item, shared across tasks and languages audio/<lang>/<id>.wav spoken question and options (Task 1a) task1a/<split>_<lang>.jsonl task1b/<split>_<lang>.jsonl ``` Media is referenced by relative path keyed on `id`, so inputs join to files directly. ### 🏷️ Fields Task 1a (`task1a/<split>_<lang>.jsonl`): | field | type | description | |---|---|---| | `id` | str | item id | | `image` | str | `images/<id>.jpg` | | `audio` | str | `audio/<lang>/<id>.wav`, the spoken question **and** the three options (no text is given; listen and answer) | | `label` | int | index (0–2) of the correct option | Task 1b (`task1b/<split>_<lang>.jsonl`): | field | type | description | |---|---|---| | `id` | str | item id | | `image` | str | `images/<id>.jpg` | | `statements` | list[str] | three statements, exactly one grounded | | `labels` | list[bool] | truth value of each statement (one `true`) | `train` and `dev` additionally include `country`, `category` and `subcategory`. These and the labels are **not** provided in `devtest`, nor will they be provided in `test`. ## 📊 Splits | split | labels | items | use | |---|---|---|---| | `train` | yes | 3000 | training and fine-tuning | | `dev` | yes | 500 | validation | | `devtest` | no | 500 | pre-competition; submit to Codabench | | `test` | no | 1000 | competition | ## 📏 Evaluation Each of the four tracks is scored separately. One **ranking metric** decides the leaderboard; the remaining columns are reported as diagnostics. A missing or unparseable prediction always counts as wrong. ### Spoken VQA A 3-way multiple choice: predict the option index `0`, `1` or `2`. | metric | role | meaning | |---|---|---| | **Accuracy** | **ranking** | fraction of items answered correctly | | Balanced accuracy | reported | mean per-class recall over the three positions | | Macro-F1 | reported | macro-averaged F1 over the three positions | ### Hallucination detection Three statements per image; predict **True/False** for each. Exactly one statement is grounded ("Q+"); the other two are hallucinated ("Q−"). | metric | role | meaning | |---|---|---| | **Contrastive Instability (CI)** | **ranking** | of the items with at least one of the three statements correct, the fraction that are not fully correct (all three right); **lower is better** | | Combined accuracy | reported | fraction of items where **all three** labels are correct (grounded → true, both hallucinated → false); higher is better | | CFHR | reported | of items where the grounded statement was correctly identified, the fraction that still affirmed a hallucinated one; **lower is better** | | Q+ accuracy | reported | grounded statement correctly marked true | | Q− accuracy | reported | hallucinated statements correctly marked false (over all false statements) | True/False is read from the prediction with the shared‑task `evaluate_tf` parser, which handles English and Arabic verdicts (e.g. `true`/`false`, صح/خطأ). ## 🧪 Baselines & example notebooks Starter **Colab notebooks** that run end to end (download the data → run a model → write a Codabench-ready `prediction.zip`) are here: **[Reference notebooks (Google Drive)](https://drive.google.com/drive/folders/1zDO2uXq0fEQhfNj8M2DstkNL1-EEXybT?usp=sharing)** - **Open-model baseline**: Task 1a (`Qwen2.5-Omni`) and Task 1b (`Qwen2.5-VL`), runnable on a free Colab T4 (4-bit). - **Cascaded API example (Task 1a)**: Fanar `Aura-STT` (speech → text) → `Oryx` (image understanding); no GPU required. These are references only; participants are free to use their own models, prompts, and configurations. Reference baseline scores (greedy decoding, `do_sample=False`, so deterministic). Contrastive instability is lower-is-better: | task | track | model | ranking metric | score | |---|---|---|---|---| | 1a | English | Qwen2.5-Omni | accuracy | 0.6640 | | 1a | MSA | Qwen2.5-Omni | accuracy | 0.3980 | | 1b | English | Qwen2.5-VL | contrastive instability | 0.3133 | | 1b | MSA | Qwen2.5-VL | contrastive instability | 0.4900 | For 1b, combined accuracy was 0.6840 (English) and 0.5080 (MSA). ## 📜 License and contact The dataset is distributed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). - Website: https://imageeval2026.github.io/ - Slack: https://join.slack.com/t/mm-eval/shared_invite/zt-44zii4e53-IiZZ3O73Za9XL4umb9mjUQ - Contact: [e-mail] ## Citation ``` @article{alam2025everydaymmqa, title = {{OASIS}: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QA}, author = {Alam, Firoj and Shahroor, Ali Ezzat and Hasan, Md. Arid and Ali, Zien Sheikh and Bhatti, Hunzalah Hassan and Kmainasi, Mohamed Bayan and Chowdhury, Shammur Absar and Mousi, Basel and Dalvi, Fahim and Durrani, Nadir and Milic-Frayling, Natasa}, journal = {arXiv preprint arXiv:2510.06371}, year = {2025}, } @inproceedings{mousi2026counterfactualhallucination, title = {Once Correct, Still Wrong: Counterfactual Hallucination in Multilingual Vision-Language Models}, author = {Mousi, Basel and Dalvi, Fahim and Chowdhury, Shammur and Alam, Firoj and Durrani, Nadir}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics}, address = {San Diego, California, United States}, publisher = {Association for Computational Linguistics}, year = {2026}, url = {https://arxiv.org/abs/2602.05437}, } @inproceedings{mousi2026said, title = {Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models}, author = {Mousi, Basel and Dalvi, Fahim and Chowdhury, Shammur and Alam, Firoj and Durrani, Nadir}, booktitle = {Proceedings of Interspeech 2026}, year = {2026}, address = {Sydney, Australia}, note = {accepted} } ```