IslamicEval2025-Subtask1B_verified

# IslamicEval 2025, Subtask 1B (dev) (verified) A filtered copy of the **IslamicEval 2025** Subtask 1B development data (hallucination detection in Islamic content: classify a span of an LLM answer as a correct or wrong Qur'anic verse or Hadith), with samples flagged by a multi-judge audit removed. One config, `default`. > **What "verified" means.** Samples were removed when all three LLM judges flagged a significant defect. This is not a human review, and only samples that baseline models answered incorrectly were audited, so some defects may remain in the kept data. ## Source - Original data: GitHub [`qcri/IslamicEval-2025-Subtask-1`](https://github.com/qcri/IslamicEval-2025-Subtask-1), commit `256e2d2d007ca5cde0f2dea6b89a45a77ee59476`, `dev_SubtaskB/dev_SubtaskB.tsv` and `.xml`. Shared task website: <https://sites.google.com/view/islamiceval-2025>. - The evaluation copy has 247 rows from 50 questions. `Question_ID`, `Label`, `Span_Start`, `Span_End` and `Original_Span` equal the TSV; `Text` and `Response` equal the XML after Unicode (NFC) normalisation. It adds a `full_span` field (the text shown to the model); this field was derived locally and is kept as is. ## Columns `Question_ID`, `Label` (`CorrectAyah`, `WrongAyah`, `CorrectHadith`, `WrongHadith`), `Span_Start`, `Span_End`, `Original_Span`, `Text` (the question), `Response` (the LLM answer) and `full_span`. The upstream `Annotation_ID` column is not included. ## Audit and filtering Samples were screened with the multi-judge audit described in [`IFM/discarded_samples`](https://huggingface.co/datasets/IFM/discarded_samples) (revision `3a3c1339`, config `IslamicEval2025-1B`). Three language-model judges (GPT-5.6 Sol, Gemini 3.8 Flash, Claude Opus 5) assessed candidate samples that strong baselines answered incorrectly. A sample is excluded only when **all three** judges report a *significant* issue (a problem with the question, the options or the gold label that prevents reliable answering or scoring). Excluded samples were matched to this benchmark by the row index (`doc_id`) assigned by lm-evaluation-harness, which equals the 0-based row index of the source file listed above; every matched row was additionally checked against the discarded record's question text and gold label. All other rows are kept unchanged and in their original order, with the original column names and values. | Config | Original rows | Excluded | Kept | |---|---:|---:|---:| | `default` | 247 | 16 | 231 | | **Total** | **247** | **16** | **231** | ## License The source repository is released under **Apache-2.0**. The data is Qur'anic and Hadith text with model-generated answers; check the repository and the shared-task terms before reuse. ## Citation Please cite the shared-task overview: ```bibtex @inproceedings{mubarak2025islamiceval, title = {IslamicEval 2025: The First Shared Task of Capturing LLMs Hallucination in Islamic Content}, author = {Mubarak, Hamdy and Malhas, Rana and Mansour, Watheq and Mohamed, Abubakr and Fawzi, Mahmoud and Hawasly, Majd and Elsayed, Tamer and Darwish, Kareem and Magdy, Walid}, booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks}, pages = {480--493}, address = {Suzhou, China}, month = {November}, year = {2025}, publisher = {Association for Computational Linguistics}, doi = {10.18653/v1/2025.arabicnlp-sharedtasks.67} } @misc{jais3_2026, title = {Jais 3: Introducing Arabic-centric Agentic AI}, author = {{IFM Team, Inception42 Team, and Cerebras Team}}, year = {2026}, url = {https://ifm.ai/blog/jais3/}, } ```