ImageEval2026-Task1

# ImageEval 2026, Task 1: Ayn-VQA 👁️ Culturally grounded Arabic multimodal evaluation, part of the [ImageEval 2026 Shared Task](https://imageeval2026.github.io/) at ArabicNLP 2026. Ayn (عين, "eye") tests whether a model can read a culturally specific image, both from a spoken Arabic question and by telling grounded descriptions apart from plausible but hallucinated ones. Each task is offered as two language tracks, **English** and **Modern Standard Arabic (MSA)**, scored separately. ## 📝 Register Please fill in the **[registration form](https://docs.google.com/forms/d/e/1FAIpQLSd1QKF4rXD_gbLJlDykLvB0DGMIogwhraeOtWRiQiotucK0zA/viewform)** — it lets the organisers keep track of participants and notify you about data releases, deadlines, and any updates. ## 🎯 Tasks **Task 1a, Spoken VQA.** Given an image and the spoken question and options (audio), choose the correct option. Prediction: the option index 0, 1 or 2. **Task 1c, Hallucination detection.** Given an image and three statements, decide for **each** statement whether it is **True** (grounded in the image) or **False** (a hallucination). Exactly one statement is grounded. Prediction: a True/False label per statement. ## 🗂️ Subsets | config | task | language | Codabench | |---|---|---|---| | `task1a_en` | Spoken VQA | English | [compete](https://www.codabench.org/competitions/17002/) | | `task1a_msa` | Spoken VQA | MSA | [compete](https://www.codabench.org/competitions/17001/) | | `task1c_en` | Hallucination | English | [compete](https://www.codabench.org/competitions/17000/) | | `task1c_msa` | Hallucination | MSA | [compete](https://www.codabench.org/competitions/16999/) | The English and MSA tracks of a task are parallel: same images, same answers, and the questions are translations of each other. ## 🌍 Countries The dataset spans **18 Arab countries**: Algeria, Bahrain, Egypt, Iraq, Jordan, Kuwait, Lebanon, Libya, Morocco, Oman, Palestine, Qatar, Saudi Arabia, Sudan, Syria, Tunisia, UAE, Yemen. ## 🔊 Audio The Task 1a questions in `train`, `dev` and `devtest` are **synthetically generated** using voice cloning (TTS). The questions in the final blind **test** set will be **human-recorded**; expect a speaker/recording-condition shift between the dev-phase audio and the test audio. ## 📂 Files ``` images/<id>.jpg one image per item, shared across tasks and languages audio/<lang>/<id>.wav spoken question and options (Task 1a) task1a/<split>_<lang>.jsonl task1c/<split>_<lang>.jsonl ``` Media is referenced by relative path keyed on `id`, so inputs join to files directly. ### 🏷️ Fields Task 1a (`task1a/<split>_<lang>.jsonl`): | field | type | description | |---|---|---| | `id` | str | item id | | `image` | str | `images/<id>.jpg` | | `audio` | str | `audio/<lang>/<id>.wav`, the spoken question **and** the three options (no text is given; listen and answer) | | `label` | int | index (0–2) of the correct option | Task 1c (`task1c/<split>_<lang>.jsonl`): | field | type | description | |---|---|---| | `id` | str | item id | | `image` | str | `images/<id>.jpg` | | `statements` | list[str] | three statements, exactly one grounded | | `labels` | list[bool] | truth value of each statement (one `true`) | `train` and `dev` additionally include `country`, `category` and `subcategory`. These and the labels are **not** provided in `devtest`, nor will they be provided in `test`. ## 📊 Splits | split | labels | items | use | |---|---|---|---| | `train` | yes | 3000 | training and fine-tuning | | `dev` | yes | 500 | validation | | `devtest` | no | 500 | pre-competition; submit to Codabench | | `test` | no | 1000 | competition | The blind **test** set is released later for the final phase. ## 🗓️ Timeline | phase | window | submit on | |---|---|---| | **Development** | `2026-05-22 → 2026-07-19` | `devtest` — leaderboard live | | **Testing** | `2026-07-20 → 2026-07-29` | `test` — blind, final ranking | Dates may shift — watch the [website](https://imageeval2026.github.io/) and the [registration form](https://docs.google.com/forms/d/e/1FAIpQLSd1QKF4rXD_gbLJlDykLvB0DGMIogwhraeOtWRiQiotucK0zA/viewform) for announcements. ## 🚀 Submitting 1. Load the subset you are targeting: ```python from datasets import load_dataset ds = load_dataset("QCRI/ImageEval2026-Task1-AynVQA", "task1c_msa", split="devtest") ``` 2. Produce predictions: - **Task 1a** — for each item, predict an index 0, 1 or 2. Write a CSV with columns `id,prediction`: ``` id,prediction 1dcdf6da...,0 803ca9b8...,2 ``` - **Task 1c** — for each item, predict True/False for each of the three statements. Write a CSV with columns `id,statement_index,prediction` (`statement_index` is 0–2; `prediction` is `true` or `false`): ``` id,statement_index,prediction 1dcdf6da...,0,false 1dcdf6da...,1,true 1dcdf6da...,2,false ``` 3. Zip the CSV as `prediction.zip` and submit to the matching Codabench competition (links in the **Subsets** table above): [task1a_en](https://www.codabench.org/competitions/17002/) · [task1a_msa](https://www.codabench.org/competitions/17001/) · [task1c_en](https://www.codabench.org/competitions/17000/) · [task1c_msa](https://www.codabench.org/competitions/16999/). ## 📏 Evaluation Each of the four tracks is scored separately. One **ranking metric** decides the leaderboard; the remaining columns are reported as diagnostics. A missing or unparseable prediction always counts as wrong. ### Task 1a — Spoken VQA A 3-way multiple choice: predict the option index `0`, `1` or `2`. | metric | role | meaning | |---|---|---| | **Accuracy** | **ranking** | fraction of items answered correctly | | Balanced accuracy | reported | mean per-class recall over the three positions | | Macro-F1 | reported | macro-averaged F1 over the three positions | ### Task 1c — Hallucination detection Three statements per image; predict **True/False** for each. Exactly one statement is grounded ("Q+"); the other two are hallucinated ("Q−"). | metric | role | meaning | |---|---|---| | **Combined accuracy** | **ranking** | fraction of items where **all three** labels are correct (grounded → true, both hallucinated → false) | | Hallucination rate | reported | `q_plus_accuracy − combined_accuracy`; **lower is better** | | Conditional hallucination rate (CFHR-2) | reported | of items where the grounded statement was correctly identified, the fraction that still affirmed a hallucinated one; **lower is better** | | CFHR-3 | reported | of items with ≥1 of the three correct, the fraction not fully correct; **lower is better** | | Q+ accuracy | reported | grounded statement correctly marked true | | Q− accuracy | reported | hallucinated statements correctly marked false (over all false statements) | | Q− both accuracy | reported | both hallucinated statements marked false | True/False is read from the prediction with the shared‑task `evaluate_tf` parser, which handles English and Arabic verdicts (e.g. `true`/`false`, صح/خطأ). ## 🧪 Baselines & reference notebooks Starter **Colab notebooks** that run end to end (download the data → run a model → write a Codabench-ready `prediction.zip`) are here: **[Reference notebooks (Google Drive)](https://drive.google.com/drive/folders/1zDO2uXq0fEQhfNj8M2DstkNL1-EEXybT?usp=sharing)** - **Open-model baseline** — Task 1a (`Qwen2.5-Omni`) and Task 1c (`Qwen2.5-VL`), runnable on a free Colab T4 (4-bit). - **Cascaded API baseline (Task 1a)** — Fanar `Aura-STT` (speech → text) → `Oryx` (image understanding); no GPU required. These are references only — participants are free to use their own models, prompts, and configurations. Reference baseline scores: | task | track | model | metric | score | |---|---|---|---|---| | 1a | English | Qwen2.5-Omni | accuracy | 0.6640 | | 1a | MSA | Qwen2.5-Omni | accuracy | 0.3980 | | 1c | English | Qwen2.5-VL | combined accuracy | 0.6840 | | 1c | MSA | Qwen2.5-VL | combined accuracy | 0.5080 | ## 📜 License and contact CC BY-NC 4.0, research use only. - Website: https://imageeval2026.github.io/ - Tasks repo: https://github.com/ImageEval2026/ImageEval2026-tasks - Contact: [e-mail]