IDiOM-QA: A Dataset for Question Answering on Office Memoranda for Indian Research and Educational Institutions

IDiOM-QA v1.0.1: full release IDiOM-QA is a question-answering benchmark over 116 Office Memoranda and related notices from four Indian academic and research bodies (UGC, CSIR, IACS, Jadavpur University). Contents data/qa/idiom_qa_v4.json: 475 QA pairs in eight types (91 Factoid, 89 Conceptual, 92 Relational, 92 Multi-part, 71 Adversarial, 16 CrossHop, 13 Temporal, 11 Aggregation), with gold answers and evidence data/corpus: hand-checked transcriptions and summaries of all 116 OMs data/kg/kg.json: typed knowledge graph (397 entities, 591 triples) data/qrels: OM-level relevance judgements for the 404 answerable questions results: per-question scores of 12 RAG systems and 2 reference runs (RAGAS, surface metrics, HHEM, AlignScore, MiniCheck, retrieval) code: IDiOM-RAG, the baselines, index and KG builders, and evaluation scripts These are the questions and results reported in the paper. Licence: data and results under CC-BY-4.0; code under MIT. The OM texts are publicly available.