BhashaBench-Multi

# BhashaBench-Multi ## Dataset Description **BhashaBench-Multi** is a large-scale multilingual multiple-choice question answering (MCQ) benchmark covering four specialised Indian knowledge domains across **22 Indian languages plus English**. No existing open-source benchmark provides purpose-built Indian domain knowledge coverage across such a broad set of languages and specialised domains. BhashaBench-Multi is designed to evaluate language understanding, domain knowledge, and cross-lingual robustness of NLP systems in low-resource and mid-resource Indian language settings. These four domains were selected as high-stakes, underserved verticals in Indian-language NLP: | Sub-dataset | Domain | |---|---| | **BhashaBench-Ayur** | Ayurveda & traditional medicine | | **BhashaBench-Finance** | Indian finance, banking & regulation | | **BhashaBench-Legal** | Indian law (IPC/BNS, Constitution, civil/commercial statutes) | | **BhashaBench-Krishi** | Agriculture, crop science & government schemes | --- ## Languages The dataset spans **22 scheduled Indian languages** as well as English: | Language | ISO Code | Script | |---|---|---| | Assamese | `as` | Bengali | | Bengali | `bn` | Bengali | | Bodo | `brx` | Devanagari | | Dogri | `doi` | Devanagari | | Gujarati | `gu` | Gujarati | | Hindi | `hi` | Devanagari | | Kannada | `kn` | Kannada | | Kashmiri | `ks` | Perso-Arabic | | Konkani | `gom` | Devanagari | | Maithili | `mai` | Devanagari | | Malayalam | `ml` | Malayalam | | Manipuri | `mni` | Meitei Mayek | | Marathi | `mr` | Devanagari | | Nepali | `ne` | Devanagari | | Odia | `or` | Odia | | Punjabi | `pa` | Gurmukhi | | Sanskrit | `sa` | Devanagari | | Santali | `sat` | Ol Chiki | | Sindhi | `sd` | Perso-Arabic | | Tamil | `ta` | Tamil | | Telugu | `te` | Telugu | | Urdu | `ur` | Perso-Arabic | | English | `en` | Latin | --- ## Dataset Structure ### Data Format Each record in the dataset contains the following fields: | Field | Type | Description | |---|---|---| | `id` | string | Unique identifier for the question | | `question` | string | Original MCQ question text | | `option_a` | string | Answer option A | | `option_b` | string | Answer option B | | `option_c` | string | Answer option C | | `option_d` | string | Answer option D | | `correct_answer` | string | Label of the correct option (`A`, `B`, `C`, or `D`) | | `question_type` | string | Type of question (e.g., `MCQ`) | | `question_level` | string | Difficulty level of the question (e.g., `Easy`, `Medium`, `Hard`) | | `topic` | string | Topic associated with the question | | `subject_domain` | string | Subject domain or category of the question | | `llm_judge_scores` | object | Quality scores assigned by an LLM judge | | `source_lang` | string | Source language code of the original question | | `target_lang` | string | Target language code of the translated question | | `question_translated` | string | Question translated into the target language | | `option_a_translated` | string | Option A translated into the target language | | `option_b_translated` | string | Option B translated into the target language | | `option_c_translated` | string | Option C translated into the target language | | `option_d_translated` | string | Option D translated into the target language | | `question_backtranslated` | string | Back-translation of the translated question into English | | `option_a_backtranslated` | string | Back-translation of option A into English | | `option_b_backtranslated` | string | Back-translation of option B into English | | `option_c_backtranslated` | string | Back-translation of option C into English | | `option_d_backtranslated` | string | Back-translation of option D into English | | `translation_quality_metrics` | object | Automatic translation quality metrics | The `llm_judge_scores` field contains the following sub-fields: | Field | Type | Description | |---|---|---| | `factual_accuracy` | float | Factual correctness score | | `question_clarity` | float | Clarity, readability, and comprehensibility score | | `option_quality` | float | Quality, plausibility, and distinctiveness of answer options | | `translation_fidelity` | float | Degree to which the evaluated English content preserves the meaning, intent, and terminology of the original source material | | `safety` | float | Safety and appropriateness score | The `translation_quality_metrics` field contains the following automatic evaluation metrics: | Field | Type | Description | |---|---|---| | `bleu` | float | BLEU score | | `chrf` | float | chrF score | | `chrf++` | float | chrF++ score | > **Field ordering convention:** All original source-language fields appear first, followed by their `_translated` counterparts. This ordering is intentional for Hugging Face compatibility. ### Dataset Statistics #### Comparison: BhashaBench V1 vs BhashaBench-Multi | Attribute | BhashaBench V1 | BhashaBench-Multi | |------------|------------|------------| | Languages | English, Hindi | 22 Indian Languages + English | | Number of Languages | 2 | 23 | | Domains | Agriculture, Finance, Law, Ayurveda | Agriculture, Finance, Law, Ayurveda | | Domain Subsets | BBK, BBF, BBL, BBA | BBK, BBF, BBL, BBA | | Original Questions | 74,166 | 74,166 | | Multilingual Expansion | No | Yes | | Translation Coverage | N/A | Full benchmark translated into 22 Indian languages | | Total Question Instances | 74,166 | 1,705,818 | | Primary Purpose | Bilingual domain knowledge evaluation | Multilingual domain knowledge evaluation | #### Original BhashaBench V1 Distribution | Language | BBK | BBF | BBA | BBL | Total | |----------|------:|------:|------:|------:|------:| | English | 12,648 | 13,451 | 9,348 | 17,047 | 52,494 | | Hindi | 2,757 | 5,982 | 5,615 | 7,318 | 21,672 | | **Total** | **15,405** | **19,433** | **14,963** | **24,365** | **74,166** | > BhashaBench-Multi is created by combining the English questions and Hindi questions translated into English from BhashaBench V1, followed by translation into 22 Indian languages. Each language contains all four domain-specific subsets (BBK, BBF, BBA, and BBL). #### Dataset Split This dataset is released as a single evaluation split with no train/test separation. --- ## Dataset Creation ### Dataset Source This dataset is derived from the **BhashaBench V1** benchmark and extends it to a multilingual setting covering 22 Indian languages and English. **BhashaBench V1 covered English and Hindi only across the same four domains: Agriculture, Finance, Law, and Ayurveda. BhashaBench-Multi extends this coverage to 22 Indian languages by applying a machine translation pipeline to the original questions and answer options.** The dataset is intended for benchmarking multilingual language models and evaluating cross-lingual knowledge transfer across specialised Indian domains. The multilingual variants in this dataset were generated by translating the original BhashaBench V1 content into multiple Indian languages. Translation quality was evaluated using automatic translation metrics, while the original English questions were evaluated using an LLM-as-a-judge framework for factual accuracy, clarity, option quality, and semantic consistency. ### Quality Evaluation Overall question quality was evaluated using an **LLM-as-a-judge** pipeline powered by **GPT-OSS-120B**. To ensure consistent evaluation across languages, all samples were evaluated in English. Questions originating from the English split of BhashaBench V1 were evaluated directly. Questions originating from the Hindi split were first translated into English and then evaluated. Importantly, the **translation_fidelity** score does not measure similarity between a translated sample and its corresponding source-language sample. Instead, it measures the extent to which the English representation used during evaluation preserves the meaning, intent, terminology, and information content of the original BhashaBench V1 source material. Consequently, for originally English samples, this score acts as a semantic consistency baseline, while for Hindi-derived samples it reflects preservation of meaning through translation. Each MCQ was scored on the following criteria: | Criterion | Description | |---|---| | `factual_accuracy` | Correctness of domain-specific facts, references, and the marked answer | | `question_clarity` | Clarity, readability, and sufficiency of context provided by the question | | `option_quality` | Quality, plausibility, and distinctiveness of the answer options | | `translation_fidelity` | Degree to which the evaluated English content preserves the meaning, intent, and terminology of the original source material | | `safety` | Absence of harmful, misleading, unsafe, or inappropriate content | ### Back-Translation Evaluation To further validate translation quality, a back-translation evaluation pipeline was performed. Translated texts were translated back into English and compared against the original source content using: - **BLEU** (sentence-level, effective order) - **chrF** - **chrF++** (word order = 2) --- ## Intended Uses - **Benchmarking** multilingual LLMs on Indian-language domain knowledge across agriculture, finance, law, and Ayurveda - **Evaluating** cross-lingual knowledge transfer and robustness across 22 Indian languages and English - **Research** on multilingual reasoning, domain adaptation, and low-resource language understanding - **Assessing** knowledge preservation and performance disparities across diverse Indian languages and scripts ## Limitations - **Machine-translated content:** The multilingual variants are generated through automated translation pipelines and may not perfectly capture all linguistic nuances, cultural context, or domain-specific terminology. - **Limited domain scope:** The benchmark focuses on four specialised domains—Agriculture, Finance, Law, and Ayurveda—and does not represent general knowledge across all subject areas. - **Multiple-choice format:** As an MCQ benchmark, it primarily evaluates answer selection and domain understanding rather than open-ended generation or complex reasoning capabilities. - **Automatic evaluation:** Translation quality metrics and LLM-as-a-judge scores provide useful quality signals but may not fully align with expert human assessment. - **Evolving knowledge:** Some content, particularly in finance, law, agriculture, and government schemes, may become outdated as regulations, policies, and domain knowledge change over time. --- ## Additional Information ### Citation BhashaBench-Multi is derived from the BhashaBench V1 benchmark. If you use this dataset in your research, please cite the original BhashaBench V1 paper: ```bibtex @misc{devane2025bhashabenchv1comprehensivebenchmark, title={BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains}, author={Vijay Devane and Mohd Nauman and Bhargav Patel and Aniket Mahendra Wakchoure and Yogeshkumar Sant and Shyam Pawar and Viraj Thakur and Ananya Godse and Sunil Patra and Neha Maurya and Suraj Racha and Nitish Kamal Singh and Ajay Nagpal and Piyush Sawarkar and Kundeshwar Vijayrao Pundalik and Rohit Saluja and Ganesh Ramakrishnan}, year={2025}, eprint={2510.25409}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2510.25409} } ``` ### Acknowledgements BhashaBench-Multi is derived from the BhashaBench V1 benchmark. Quality evaluation was performed using an LLM-as-a-judge framework based on Param-2-17B-A2.4B and [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b). ### Licensing This dataset is released under the CC BY 4.0 license. You are free to share and adapt the material for any purpose, provided appropriate attribution is given to the original authors.