LingoIITGN/PI-Indic-Align

---
language:
- as
- bn
- gu
- hi
- kn
- ml
- mr
- or
- pa
- ta
- te
- ur
license: cc-by-4.0
multilinguality:
- multilingual
size_categories:
- 100K<n<1M
task_categories:
- text-retrieval
- text-classification
- sentence-similarity
task_ids:
- semantic-similarity-scoring
- semantic-similarity-classification
pretty_name: "PI-Indic-Align: Persona-Instruction Alignment for Indian Languages"
tags:
- persona-instruction-alignment
- indian-languages
- cross-lingual-retrieval
- multilingual-embeddings
- benchmark
- indic-nlp
configs:
- config_name: default
data_files:
- "nirdesha_asm_Beng.jsonl"
- "nirdesha_ben_Beng.jsonl"
- "nirdesha_guj_Gujr.jsonl"
- "nirdesha_hin_Deva.jsonl"
- "nirdesha_kan_Knda.jsonl"
- "nirdesha_mal_Mlym.jsonl"
- "nirdesha_mar_Deva.jsonl"
- "nirdesha_ory_Orya.jsonl"
- "nirdesha_pan_Guru.jsonl"
- "nirdesha_tam_Taml.jsonl"
- "nirdesha_tel_Telu.jsonl"
- "nirdesha_urd_Arab.jsonl"
- config_name: assamese
data_files: "nirdesha_asm_Beng.jsonl"
- config_name: bengali
data_files: "nirdesha_ben_Beng.jsonl"
- config_name: gujarati
data_files: "nirdesha_guj_Gujr.jsonl"
- config_name: hindi
data_files: "nirdesha_hin_Deva.jsonl"
- config_name: kannada
data_files: "nirdesha_kan_Knda.jsonl"
- config_name: malayalam
data_files: "nirdesha_mal_Mlym.jsonl"
- config_name: marathi
data_files: "nirdesha_mar_Deva.jsonl"
- config_name: odia
data_files: "nirdesha_ory_Orya.jsonl"
- config_name: punjabi
data_files: "nirdesha_pan_Guru.jsonl"
- config_name: tamil
data_files: "nirdesha_tam_Taml.jsonl"
- config_name: telugu
data_files: "nirdesha_tel_Telu.jsonl"
- config_name: urdu
data_files: "nirdesha_urd_Arab.jsonl"
---

# PI-Indic-Align: Persona-Instruction Alignment for Indian Languages

<div align="center">
<img src="./assets/mascot.jpg" alt="PI-Indic-Align Mascot" width="300"/>

**🦚 PI-Indic-Align: Persona-Instruction Alignment for Indian Languages**

<em>Teaching AI to speak the languages of India, one persona at a time!</em>
</div>

## Dataset Description

**PI-Indic-Align** is a large-scale benchmark dataset for evaluating persona-instruction alignment across 12 major Indian languages. The dataset contains 600,000 culturally grounded persona-instruction pairs (50,000 per language) designed to assess how well multilingual embedding models capture compatibility between user personas and instructions in Indic languages. [ppl-ai-file-upload.s3.amazonaws](https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/104192482/a1adbddc-b3aa-4def-b574-5dc8cf0e698b/One_Instruction_Does_Not_Fit_All_ACL.pdf)

### Key Features

- **12 Indian Languages**: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu, and Urdu
- **600,000 Total Pairs**: 50,000 persona-instruction pairs per language
- **Cross-lingual Aligned**: Shared identifiers enable cross-lingual evaluation across different scripts
- **Human Validated**: 1,800 pairs validated by native speakers with substantial to almost perfect inter-annotator agreement (Cohen's κ: 0.63-0.84)
- **Four Evaluation Tasks**: Monolingual retrieval, cross-lingual retrieval, reverse retrieval, and binary compatibility classification

### Languages

The dataset covers two major language families:
- **Indo-Aryan**: Assamese, Bengali, Gujarati, Hindi, Marathi, Odia, Punjabi, Urdu
- **Dravidian**: Kannada, Malayalam, Tamil, Telugu

These languages employ 10 distinct scripts, making the dataset ideal for evaluating cross-script transfer capabilities.

## Dataset Structure

### Data Fields

Each entry contains:
- `indian_context_persona`: A persona description specifying demographic, occupational, or cultural attributes relevant to Indian users
- `indian_context_instruction`: A compatible instruction that the persona might plausibly issue to an AI assistant
- `language`: ISO 639-3 language code
- `pair_id`: Unique identifier enabling cross-lingual alignment

### Data Splits

The dataset includes development and test splits for evaluation purposes, designed for four benchmark tasks:

1. **T1 - Monolingual Retrieval**: Retrieve compatible instruction given a persona in the same language
2. **T2 - Cross-Lingual Retrieval**: Retrieve compatible instruction in a different language
3. **T3 - Reverse Retrieval**: Retrieve compatible persona given an instruction (both monolingual and cross-lingual)
4. **T4 - Compatibility Classification**: Binary classification of persona-instruction compatibility

### Example

```python
{
"indian_context_persona": "एक थिएटर प्रबंधक या इवेंट समन्वयक जो मुंबई में एनसीपीए जैसे प्रदर्शन कला केंद्रों की परिचालन पहलुओं, सुविधाओं और प्रोग्रामिंग को समझने में रुचि रखता है।",
"indian_context_instruction": "मुंबई में एनसीपीए जैसे प्रदर्शन कला केंद्र में बहु-शो महोत्सव की मेजबानी करते समय शेड्यूलिंग और रसद के लिए प्रमुख विचार क्या हैं?",
"language": "hin",
"pair_id": "example_001"
}
```

## Dataset Creation

### Source Data

#### Synthesis Process

1. **Persona Structuring**: Following PersonaHub principles, 50,000 English persona-instruction pairs were synthesized using GPT-4o-mini
2. **Domain Coverage**: Pairs span education, commerce, healthcare, agriculture, and government services to ensure diversity
3. **Cultural Grounding**: All personas and instructions are contextualized for Indian users

#### Translation

- **Model**: NLLB-200 (3.3B parameters) for high-quality multilingual translation
- **Normalization**: Script-specific punctuation and whitespace conventions applied
- **Alignment**: Shared identifiers maintain cross-lingual correspondence

### Human Validation

- **Sample Size**: 150 pairs per language (1,800 total)
- **Annotators**: Two independent native speakers per language
- **Agreement Metrics**: Cohen's κ (0.63-0.84), Krippendorff's α, Gwet's AC1, and percent agreement (88.7%-96.7%)
- **Task**: Judge whether translated pairs preserve compatibility relationships from English originals

## Use Cases

### Primary Applications

1. **Embedding Model Evaluation**: Benchmark multilingual sentence embeddings for persona-instruction retrieval
2. **Cross-Lingual Transfer Analysis**: Assess model performance across different scripts and language families
3. **Personalized AI Assistants**: Train and evaluate persona-aware instruction matching systems
4. **Cultural Alignment Research**: Study how well models capture culturally grounded user preferences

### Model Benchmarking

The dataset enables evaluation of:
- Multilingual sentence encoders (BERT-based, instruction-tuned, etc.)
- Cross-lingual retrieval systems
- Persona-grounded dialogue systems
- Semantic similarity models for low-resource languages

## Benchmark Results

### Top-Performing Models

| Task | Best Model | Score |
|------|-----------|-------|
| T1 Monolingual (R@1) | E5-Large-Instruct | 27.4% |
| T2 Cross-lingual (R@1) | E5-Large-Instruct | 20.7% |
| T3 Reverse Mono (R@1) | BGE-M3 | 32.1% |
| T3 Reverse Cross (R@1) | E5-Large-Instruct | 27.0% |
| T4 Classification (AUROC) | LaBSE | 75.3% |

See the paper for complete results across 8 multilingual embedding models.

## Citation

If you use this dataset, please cite:

```bibtex
@misc{shah2026instructiondoesfitall,
title={One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages?},
author={Arya Shah and Himanshu beniwal and Mayank Singh},
year={2026},
eprint={2601.10205},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2601.10205},
}
```

## Additional Information

### Dataset Curators

LingoIITGN Research Group, IIT Gandhinagar

### Licensing Information

[Please specify your license - e.g., CC-BY-4.0, MIT, Apache 2.0]

### Contributions

Contributions and feedback are welcome! Please visit our [GitHub repository](https://github.com/aryashah2k/PI-Indic-Align) to:
- Report issues
- Submit pull requests
- Access evaluation scripts
- View baseline implementations

### Contact

For questions or collaborations, please open an issue on the [GitHub repository](https://github.com/aryashah2k/PI-Indic-Align).

### Acknowledgments

This dataset was created using:
- GPT-4o-mini for persona-instruction synthesis
- NLLB-200 (3.3B) for translation
- PersonaHub structuring principles

Special thanks to all native-speaker annotators who contributed to human validation.