teias-ai/percul

---
language:
- fa
size_categories:
- n<1K
task_categories:
- question-answering
pretty_name: PerCul
dataset_info:
features:
- name: ID
dtype: string
- name: Category
dtype: string
- name: Topic
dtype: string
- name: Story
dtype: string
- name: Choice 1
dtype: string
- name: Choice 2
dtype: string
- name: Choice 3
dtype: string
- name: Choice 4
dtype: string
- name: Correct Choice
dtype: int64
splits:
- name: original_persian
num_bytes: 647157
num_examples: 592
- name: gpt_english_translation
num_bytes: 446913
num_examples: 592
download_size: 557876
dataset_size: 1094070
configs:
- config_name: default
data_files:
- split: original_persian
path: data/original_persian-*
- split: gpt_english_translation
path: data/gpt_english_translation-*
tags:
- Persian_Benchmarking
- Culture
- Persian
- Farsi
- Benchmark
---
# Dataset Card for Dataset Name
ᴘᴇʀᴄᴜʟ (PerCul) is a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture.
### Dataset Description

Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLMs are increasingly used across diverse contexts without adequate evaluation of their cultural competence in non-English languages, including Persian. To address this gap, we introduce PerCul, a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture. PerCul features story-based, multiple-choice questions that capture culturally nuanced scenarios. Unlike existing benchmarks, PerCul is curated with input from native Persian annotators to ensure authenticity and to prevent the use of translation as a shortcut.

- **Curated by:** Joint work of [Erfan Moosavi Monazzah](https://huggingface.co/ErfanMoosaviMonazzah) & [Vahid Rahimzadeh](https://huggingface.co/vahyd)
- **Funded by:** [Tehran Institute for Advanced Studies (TeIAS)](https://teias.institute/)
- **Shared by:** [LLMs Lab @ TeIAS](https://teias.ai/)
- **Language(s) (NLP):** Persian (FA)

### Dataset Sources [optional]

<!-- Provide the basic links for the dataset. -->

- **Repository:** [teias-ai/percul](https://huggingface.co/datasets/teias-ai/percul)
- **Paper:** [ACL Anthology](https://aclanthology.org/2025.naacl-long.631/) | [arXiv](https://arxiv.org/abs/2502.07459)

## Uses

<!-- Address questions around how the dataset is intended to be used. -->
This dataset is intented to be used as a benchmark of how well your LLM understand Persian culture.

## Dataset Structure
This dataset has the following splits:
| split name | Description |
|-------------------------|----------------------------------------------------------------------------------------------|
| original_persian | The Original Curated Data |
| gpt_english_translation | The English Translation* of the Data used in the ***Impact of Translation*** Section of the Paper |

A sample row of the dataset:
```python
{
'ID': '0_O_OBJ', <str>
'Category': 'objects', <str>
'Topic': 'آفتابه', <str>
'Story': 'متن داستان در اینجا قرار دارد', <str>
'Choice 1': 'آفتابه', <str>
'Choice 2': 'پمپ آب', <str>
'Choice 3': 'شلنگ آب', <str>
'Choice 4': 'ابزار باغبانی', <str>
'Correct Choice': 1 <int>
}
```
Dataset consists of the following categories:
| English | Persian | Code Name in Dataset | # Samples |
|------------------|-----------------------------------|------------------|-----------|
| Foods | غذا، دسر و خوراکی | foods | 191 |
| Visible Behavior | رفتارهای قابل مشاهده | visible_behavior | 56 |
| Iconic Figures | اشخاص معروف و مهم | iconic_figures | 55 |
| Institutions | اماکن و نهادها | institution | 43 |
| Architecture | معماری | architecture | 43 |
| Objects | اشیا | objects | 42 |
| Appropriacy | هنجارهای اجتماعی | appropriacy | 36 |
| Dress | لباس، جواهرات و لوازم آرایشی | dress | 33 |
| Music | موسیقی ایرانی و موارد مربوط به آن | music | 32 |
| Art | هنر ایرانی و موارد مربوط به آن | art | 32 |
| Rituals | مراسم‌ها | rituals | 29 |

## Citation

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

```
@inproceedings{moosavi-monazzah-etal-2025-percul,
title = "{P}er{C}ul: A Story-Driven Cultural Evaluation of {LLM}s in {P}ersian",
author = "Moosavi Monazzah, Erfan and
Rahimzadeh, Vahid and
Yaghoobzadeh, Yadollah and
Shakery, Azadeh and
Pilehvar, Mohammad Taher",
editor = "Chiruzzo, Luis and
Ritter, Alan and
Wang, Lu",
booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
month = apr,
year = "2025",
address = "Albuquerque, New Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.naacl-long.631/",
pages = "12670--12687",
ISBN = "979-8-89176-189-6",
abstract = "Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLMs are increasingly used across diverse contexts without adequate evaluation of their cultural competence in non-English languages, including Persian. To address this gap, we introduce PerCul, a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture. PerCul features story-based, multiple-choice questions that capture culturally nuanced scenarios.Unlike existing benchmarks, PerCul is curated with input from native Persian annotators to ensure authenticity and to prevent the use of translation as a shortcut. We evaluate several state-of-the-art multilingual and Persian-specific LLMs, establishing a foundation for future research in cross-cultural NLP evaluation. Our experiments demonstrate a 11.3{\%} gap between best closed source model and layperson baseline while the gap increases to 21.3{\%} by using the best open-weight model. You can access the dataset from here:https://huggingface.co/datasets/teias-ai/percul"
}
```