bharatgenai/IndicParam

---
configs:
- config_name: IndicParam
data_files:
- path: data*
split: test
tags:
- benchmark
- low-resource
- indic-languages
task_categories:
- question-answering
- text-classification
license: cc-by-nc-4.0
language:
- npi
- guj
- mar
- ory
- doi
- mai
- san
- brx
- sat
- gom
---

## Dataset Card for IndicParam

[Paper](https://arxiv.org/abs/2512.00333) | [Code](https://github.com/ayushbits/IndicParam)

### Dataset Summary

IndicParam is a graduate-level benchmark designed to evaluate Large Language Models (LLMs) on their understanding of **low- and extremely low-resource Indic languages**.
The dataset contains **13,207 multiple-choice questions (MCQs)** across **11 Indic languages**, plus a separate **Sanskrit–English code-mixed** set, all sourced from official UGC-NET language question papers and answer keys.

### Supported Tasks

- **`multiple-choice-qa`**: Evaluate LLMs on graduate-level multiple-choice question answering across low-resource Indic languages.
- **`language-understanding-evaluation`**: Assess language-specific competence (morphology, syntax, semantics, discourse) using explicitly labeled questions.
- **`general-knowledge-evaluation`**: Measure factual and domain knowledge in literature, culture, history, and related disciplines.
- **`question-type-evaluation`**: Analyze performance across MCQ formats (Normal MCQ, Assertion–Reason, List Matching, etc.).

### Languages

IndicParam covers the following languages and one code-mixed variant:

- **Low-resource (4)**: Nepali, Gujarati, Marathi, Odia
- **Extremely low-resource (7)**: Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, Konkani
- **Code-mixed**: Sanskrit–English (Sans-Eng)

Scripts:

- **Devanagari**: Nepali, Marathi, Maithili, Konkani, Bodo, Dogri, Rajasthani, Sanskrit
- **Gujarati**: Gujarati
- **Odia (Orya)**: Odia
- **Ol Chiki (Olck)**: Santali

All questions are presented in the **native script** of the target language (or in code-mixed form for Sans-Eng).

---

## Dataset Structure

### Data Instances

Each instance is a single MCQ from a UGC-NET language paper. An example (Maithili):

```json
{
"unique_question_id": "782166eef1efd963b5db0e8aa42b9a6e",
"subject": "Maithili",
"exam_name": "Question Papers of NET Dec. 2012 Maithili Paper III hindi",
"paper_number": "Question Papers of NET Dec. 2012 Maithili Paper III hindi",
"question_number": 1,
"question_text": "मिथिलाभाषा रामायण' में सीताराम-विवाहक वर्णन भेल अछि -",
"option_a": "बालकाण्डमें",
"option_b": "अयोध्याकाण्डमे",
"option_c": "सुन्दरकाण्डमे",
"option_d": "उत्तरकाण्डमे",
"correct_answer": "a",
"question_type": "Normal MCQ"
}
```

Questions span:

- **Language Understanding (LU)**: linguistics and grammar (phonology, morphology, syntax, semantics, discourse).
- **General Knowledge (GK)**: literature, authors, works, cultural concepts, history, and related factual content.

### Data Fields

- **`unique_question_id`** *(string)*: Unique identifier for each question.
- **`subject`** *(string)*: Name of the language / subject (e.g., `Nepali`, `Maithili`, `Sanskrit`).
- **`exam_name`** *(string)*: Full exam name (UGC-NET session and subject).
- **`paper_number`** *(string)*: Paper identifier as given by UGC-NET.
- **`question_number`** *(int)*: Question index within the original paper.
- **`question_text`** *(string)*: Question text in the target language (or Sanskrit–English code-mixed).
- **`option_a`**, **`option_b`**, **`option_c`**, **`option_d`** *(string)*: Four answer options.
- **`correct_answer`** *(string)*: Correct option label (`a`, `b`, `c`, or `d`).
- **`question_type`** *(string)*: Question format, one of:
- `Normal MCQ`
- `Assertion and Reason`
- `List Matching`
- `Fill in the blanks`
- `Identify incorrect statement`
- `Ordering`

### Data Splits

IndicParam is provided as a **single evaluation split**:

| Split | Number of Questions |
| ----- | ------------------- |
| test | 13,207 |

All rows are intended for **evaluation only** (no dedicated training/validation splits).

---

## Language Distribution

The benchmark follows the distribution reported in the IndicParam paper:

| Language | #Questions | Script | Code |
| ------------- | ---------- | -------- | ---- |
| Nepali | 1,038 | Devanagari | npi |
| Marathi | 1,245 | Devanagari | mar |
| Gujarati | 1,044 | Gujarati | guj |
| Odia | 577 | Orya | ory |
| Maithili | 1,286 | Devanagari | mai |
| Konkani | 1,328 | Devanagari | gom |
| Santali | 873 | Olck | sat |
| Bodo | 1,313 | Devanagari | brx |
| Dogri | 1,027 | Devanagari | doi |
| Rajasthani | 1,190 | Devanagari | – |
| Sanskrit | 1,315 | Devanagari | san |
| Sans-Eng | 971 | (code-mixed) | – |
| **Total** | **13,207** | | |

Each language’s questions are drawn from its respective UGC-NET language papers.

---

## Dataset Creation

### Source and Collection

- **Source**: Official UGC-NET language question papers and answer keys, downloaded from the UGC-NET/NTA website.
- **Scope**: Multiple exam sessions and years, covering language/literature and linguistics papers for each of the 11 languages plus the Sanskrit–English code-mixed set.
- **Extraction**:
- Machine-readable PDFs are parsed directly.
- Non-selectable PDFs are processed using OCR.
- All text is normalized while preserving the original script and content.

### Annotation

In addition to the raw MCQs, each question is annotated by question type (described in detail in the paper):

- **Question type**:
- Multiple-choice, Assertion–Reason, List Matching, Fill in the blanks, Identify incorrect statement, Ordering.

These annotations support fine-grained analysis of model behavior across **knowledge vs. language ability** and **question format**.

---

## Sample Usage

The GitHub repository provides several Python scripts to evaluate models on the IndicParam dataset. You can adapt these scripts for your specific use case.

Typical usage pattern, as described in the GitHub README:

- **Prepare environment**: Install Python dependencies (see `requirements.txt` if present in the GitHub repository) and configure any required API keys or model caches.
- **Run evaluation**: Invoke one of the scripts with your chosen model configuration and an output directory; the scripts will:
- Load `data.csv`
- Construct language-aware MCQ prompts
- Record model predictions and compute accuracy

Scripts available in the [GitHub repository](https://github.com/ayushbits/IndicParam):
- `evaluate_open_models.py`: Example script to evaluate open-weight Hugging Face models on IndicParam.
- `evaluate_gpt_oss.py`: script to run the GPT-OSS-120B model on the same data.
- `evaluate_openrouter.py`: script to benchmark closed models via the OpenRouter API.

Script-level arguments and options are documented via the `-h`/`--help` flags within each script.

```bash
# Example of running evaluation with an open-weight model:
python evaluate_open_models.py --model_name_or_path google/gemma-2b --output_dir results/gemma-2b

# Example of running evaluation with GPT-OSS:
python evaluate_gpt_oss.py --model_name_or_path openai/gpt-oss-120b --output_dir results/gpt-oss-120b
```

---

## Considerations for Using the Data

### Social Impact

IndicParam is designed to:

- Enable rigorous evaluation of LLMs on **under-represented Indic languages** with substantial speaker populations but very limited web presence.
- Encourage **culturally grounded** AI systems that perform robustly on Indic scripts and linguistic phenomena.
- Highlight the performance gaps between high-resource and low-/extremely low-resource Indic languages, informing future pretraining and data collection efforts.

Users should be aware that the content is drawn from **academic examinations**, and may over-represent formal, exam-style language relative to everyday usage.

### Evaluation Guidelines

To align with the paper and allow consistent comparison:

1. **Task**: Treat each instance as a multiple-choice QA item with four options.
2. **Input format**: Present `question_text` plus the four options (`A–D`) to the model.
3. **Required output**: A single option label (`A`, `B`, `C`, or `D`), with no explanation.
4. **Decoding**: Use **greedy decoding / temperature = 0 / `do_sample = False`** to ensure deterministic outputs.
5. **Metric**: Compute **accuracy** based on exact match between predicted option and `correct_answer` (case-insensitive after mapping to A–D).
6. **Analysis**:
- Report **overall accuracy**.
- Break down results **per language**.

---

## Additional Information

### Citation Information

If you use IndicParam in your research, please cite:
```bibtex

@misc{maheshwari2025indicparambenchmarkevaluatellms,
title={IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages},
author={Ayush Maheshwari and Kaushal Sharma and Vivek Patel and Aditya Maheshwari},
year={2025},
eprint={2512.00333},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.00333},
}
```

### License
CCbyNC
IndicParam is released for **non-commercial research and evaluation**

### Acknowledgments

IndicParam was curated and annotated by the authors and native-speaker annotators as described in the paper.
We acknowledge UGC-NET/NTA for making examination materials publicly accessible, and the broader Indic NLP community for foundational tools and resources.

---
配置项：
- 配置名称：IndicParam
数据文件：
- 路径：data*
划分：测试集（test）
标签：
- 基准测试数据集（benchmark）
- 低资源语言（low-resource）
- 印度本土语言（indic-languages）
任务类别：
- 问答任务（question-answering）
- 文本分类任务（text-classification）
许可协议：cc-by-nc-4.0
涉及语言：
- npi
- guj
- mar
- ory
- doi
- mai
- san
- brx
- sat
- gom
---

## IndicParam 数据集卡片

[论文](https://arxiv.org/abs/2512.00333) | [代码](https://github.com/ayushbits/IndicParam)

### 数据集概述

IndicParam 是一款面向研究生层级的基准测试数据集，旨在评估大语言模型（Large Language Models, LLMs）对**低资源及极端低资源印度本土语言**的理解能力。
该数据集包含覆盖**11种印度本土语言**的**13207道多项选择题（Multiple-Choice Questions, MCQs）**，以及一套独立的**梵语-英语语码混合**试题集，所有试题均源自官方印度大学拨款委员会-国家资格考试（University Grants Commission-National Eligibility Test, UGC-NET）语言类考卷及参考答案。

### 支持任务

- **`多项选择问答（multiple-choice-qa）`**：评估大语言模型在低资源印度本土语言上完成研究生层级多项选择问答的能力。
- **`语言理解评估（language-understanding-evaluation）`**：通过带显式标注的试题，评估模型针对特定语言的语言能力（包括词法、句法、语义及语篇层面）。
- **`通用知识评估（general-knowledge-evaluation）`**：衡量模型在文学、文化、历史及相关学科领域的事实性与专业性知识掌握情况。
- **`试题类型评估（question-type-evaluation）`**：分析模型在不同多项选择题型（常规多项选择题、断言-推理题、列表匹配题等）上的表现。

### 涉及语言

IndicParam 涵盖以下11种语言及1套语码混合变体：

- **低资源语言（4种）**：尼泊尔语（Nepali）、古吉拉特语（Gujarati）、马拉地语（Marathi）、奥里亚语（Odia）
- **极端低资源语言（7种）**：多格里语（Dogri）、迈蒂利语（Maithili）、拉贾斯坦语（Rajasthani）、梵语（Sanskrit）、博多语（Bodo）、桑塔利语（Santali）、孔卡尼语（Konkani）
- **语码混合集**：梵语-英语（Sans-Eng）

书写系统：

- **天城文（Devanagari）**：尼泊尔语、马拉地语、迈蒂利语、孔卡尼语、博多语、多格里语、拉贾斯坦语、梵语
- **古吉拉特文（Gujarati）**：古吉拉特语
- **奥里亚文（Odia/Orya）**：奥里亚语
- **奥尔奇基文（Ol Chiki, Olck）**：桑塔利语

所有试题均以目标语言的**原生书写系统**呈现（梵语-英语混合集则采用语码混合形式）。

---

## 数据集结构

### 数据实例

每个数据实例均为一道源自UGC-NET语言类考卷的多项选择题。以下为迈蒂利语试题示例：

json
{
"unique_question_id": "782166eef1efd963b5db0e8aa42b9a6e",
"subject": "Maithili",
"exam_name": "Question Papers of NET Dec. 2012 Maithili Paper III hindi",
"paper_number": "Question Papers of NET Dec. 2012 Maithili Paper III hindi",
"question_number": 1,
"question_text": "मिथिलाभाषा रामायण' में सीताराम-विवाहक वर्णन भेल अछि -",
"option_a": "बालकाण्डमें",
"option_b": "अयोध्याकाण्डमे",
"option_c": "सुन्दरकाण्डमे",
"option_d": "उत्तरकाण्डमे",
"correct_answer": "a",
"question_type": "Normal MCQ"
}

试题覆盖范畴：

- **语言理解（LU）**：语言学与语法知识（涵盖音系学、词法、句法、语义学及语篇分析）
- **通用知识（GK）**：文学、作家、作品、文化概念、历史及相关事实性内容

### 数据字段

- **`unique_question_id`** *(字符串类型)*：每道试题的唯一标识符
- **`subject`** *(字符串类型)*：语言/学科名称（例如`Nepali`、`Maithili`、`Sanskrit`）
- **`exam_name`** *(字符串类型)*：完整考试名称（包含UGC-NET考试场次与对应学科）
- **`paper_number`** *(字符串类型)*：UGC-NET官方给出的考卷编号
- **`question_number`** *(整数类型)*：原考卷内的试题序号
- **`question_text`** *(字符串类型)*：目标语言的试题文本（梵语-英语混合集则为语码混合形式）
- **`option_a`**、**`option_b`**、**`option_c`**、**`option_d`** *(字符串类型)*：四个候选答案
- **`correct_answer`** *(字符串类型)*：正确选项的标签（`a`、`b`、`c`或`d`）
- **`question_type`** *(字符串类型)*：试题题型，可选值包括：
- `常规多项选择题（Normal MCQ）`
- `断言-推理题（Assertion and Reason）`
- `列表匹配题（List Matching）`
- `填空题（Fill in the blanks）`
- `识别错误表述题（Identify incorrect statement）`
- `排序题（Ordering）`

### 数据划分

IndicParam 仅提供**单一评估划分**：

| 划分名称 | 试题数量 |
| ------ | -------- |
| 测试集（test） | 13207 |

所有数据行仅用于**评估用途**，未设置专门的训练集与验证集划分。

---

## 语言分布

该基准测试数据集的语言分布与IndicParam论文中公布的一致：

| 语言名称 | 试题数量 | 书写系统 | 语言代码 |
| ------ | -------- | ------ | ------ |
| 尼泊尔语（Nepali） | 1038 | 天城文（Devanagari） | npi |
| 马拉地语（Marathi） | 1245 | 天城文（Devanagari） | mar |
| 古吉拉特语（Gujarati） | 1044 | 古吉拉特文（Gujarati） | guj |
| 奥里亚语（Odia） | 577 | 奥里亚文（Orya） | ory |
| 迈蒂利语（Maithili） | 1286 | 天城文（Devanagari） | mai |
| 孔卡尼语（Konkani） | 1328 | 天城文（Devanagari） | gom |
| 桑塔利语（Santali） | 873 | 奥尔奇基文（Olck） | sat |
| 博多语（Bodo） | 1313 | 天城文（Devanagari） | brx |
| 多格里语（Dogri） | 1027 | 天城文（Devanagari） | doi |
| 拉贾斯坦语（Rajasthani） | 1190 | 天城文（Devanagari） | – |
| 梵语（Sanskrit） | 1315 | 天城文（Devanagari） | san |
| 梵语-英语混合集（Sans-Eng） | 971 | （语码混合） | – |
| **总计** | **13207** | | |

各语言的试题均源自对应语言的UGC-NET官方考卷。

---

## 数据集构建

### 来源与采集

- **数据来源**：从UGC-NET/NTA官方网站下载的UGC-NET语言类考卷及参考答案
- **采集范围**：涵盖多场次、多年份的11种印度本土语言及梵语-英语混合集的语言/文学与语言学类考卷
- **文本提取**：
- 可直接解析的机器可读PDF将直接提取文本
- 不可选中文本的PDF将通过光学字符识别（Optical Character Recognition, OCR）进行处理
- 所有文本均在保留原始书写系统与内容的前提下进行规范化处理

### 标注流程

除原始多项选择题外，每道试题均会标注对应的题型（详细说明见论文）：
- **题型标注**：包括常规多项选择题、断言-推理题、列表匹配题、填空题、识别错误表述题、排序题
此类标注支持对模型行为进行细粒度分析，包括**知识掌握与语言能力的对比**以及**不同试题题型的表现差异**。

---

## 使用示例

GitHub代码仓库中提供了多款Python脚本，用于在IndicParam数据集上评估大语言模型，用户可根据自身需求对脚本进行调整。

典型使用流程（详见GitHub仓库的README文件）：

- **配置运行环境**：安装Python依赖包（若仓库中存在`requirements.txt`请参照其内容），并配置所需的API密钥或模型缓存
- **运行评估**：通过指定的模型配置与输出目录调用其中一款脚本，脚本将完成以下操作：
- 加载`data.csv`数据文件
- 构建适配目标语言的多项选择题提示词
- 记录模型预测结果并计算准确率

仓库中提供的脚本包括：
- `evaluate_open_models.py`：用于评估Hugging Face开源权重模型的示例脚本
- `evaluate_gpt_oss.py`：用于在同一数据集上运行GPT-OSS-120B模型的脚本
- `evaluate_openrouter.py`：通过OpenRouter API对闭源模型进行基准测试的脚本

各脚本的参数与选项可通过`-h`/`--help`命令查看帮助文档。

bash
# 示例：使用开源权重模型运行评估
python evaluate_open_models.py --model_name_or_path google/gemma-2b --output_dir results/gemma-2b

# 示例：使用GPT-OSS运行评估
python evaluate_gpt_oss.py --model_name_or_path openai/gpt-oss-120b --output_dir results/gpt-oss-120b

---

## 数据集使用注意事项

### 社会影响

IndicParam 的设计目标包括：
- 实现对**使用人口众多但网络资源极为有限的代表性不足的印度本土语言**上的大语言模型进行严谨评估
- 推动构建**具备文化适配性**的人工智能系统，使其能够在印度本土书写系统与语言现象上表现稳健
- 凸显高资源语言与低/极端低资源印度本土语言之间的模型性能差距，为未来的预训练与数据采集工作提供参考
用户需注意：本数据集内容源自**学术考试**，相较于日常使用场景，其可能过度侧重正式的考试型语言表达。

### 评估指南

为与论文标准保持一致并确保评估结果可复现与可比，请遵循以下指南：
1. **任务设定**：将每个数据实例视为一道包含四个候选选项的多项选择题问答任务
2. **输入格式**：向模型提供`question_text`文本与四个候选选项（`A–D`）
3. **输出要求**：仅输出单个正确选项的标签（`A`、`B`、`C`或`D`），无需额外解释
4. **解码策略**：使用**贪心解码/温度系数=0/`do_sample = False`**以确保输出结果确定可控
5. **评估指标**：基于预测选项与`correct_answer`的精确匹配计算**准确率**（将标签映射为A-D后不区分大小写）
6. **结果分析**：
- 报告整体准确率
- 按语言分类拆解评估结果

---

## 补充信息

### 引用信息

若您在研究中使用IndicParam数据集，请引用以下文献：
bibtex

@misc{maheshwari2025indicparambenchmarkevaluatellms,
title={IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages},
author={Ayush Maheshwari and Kaushal Sharma and Vivek Patel and Aditya Maheshwari},
year={2025},
eprint={2512.00333},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.00333},
}

### 许可协议

许可协议：CC BY-NC 4.0
IndicParam 仅用于**非商业性研究与评估**。

### 致谢

IndicParam 由论文作者与母语标注人员进行整理与标注，具体流程详见论文。
我们感谢UGC-NET/NTA将考试资料公开上线，同时感谢整个印度自然语言处理（Indic NLP）社区提供的基础工具与资源支持。