QCRI/AraDiCE-PIQA

---
license: cc-by-nc-sa-4.0
dataset_info:
- config_name: PIQA-eng
splits:
- name: test
num_examples: 1838
- config_name: PIQA-msa
splits:
- name: test
num_examples: 1838
- config_name: PIQA-lev
splits:
- name: test
num_examples: 1838
- config_name: PIQA-egy
splits:
- name: test
num_examples: 1838
configs:
- config_name: PIQA-eng
data_files:
- split: test
path: PIQA_eng/validation.json
- config_name: PIQA-msa
data_files:
- split: test
path: PIQA_msa/validation.json
- config_name: PIQA-lev
data_files:
- split: test
path: PIQA_lev/validation.json
- config_name: PIQA-egy
data_files:
- split: test
path: PIQA_egy/validation.json
---

# AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs

## Overview

The **AraDiCE** dataset is designed to evaluate dialectal and cultural capabilities in large language models (LLMs). The dataset consists of post-edited versions of various benchmark datasets, curated for validation in cultural and dialectal contexts relevant to Arabic. In this repository we show the PIQA split of the data

<!-- ## File/Directory

TO DO:

- **licenses_by-nc-sa_4.0_legalcode.txt** License information.
- **README.md** This file. -->

## Evaluation
We have used [lm-harness](https://github.com/EleutherAI/lm-evaluation-harness) eval framework to for the benchmarking. We will soon release them. Stay tuned!!

## License

The dataset is distributed under the **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0)**. The full license text can be found in the accompanying `licenses_by-nc-sa_4.0_legalcode.txt` file.

## Citation
Please find the paper <a href="https://arxiv.org/pdf/2409.11404" target="_blank" style="margin-right: 15px; margin-left: 10px">here.</a>

```
@article{mousi2024aradicebenchmarksdialectalcultural,
title={{AraDiCE}: Benchmarks for Dialectal and Cultural Capabilities in LLMs},
author={Basel Mousi and Nadir Durrani and Fatema Ahmad and Md. Arid Hasan and Maram Hasanain and Tameem Kabbani and Fahim Dalvi and Shammur Absar Chowdhury and Firoj Alam},
year={2024},
publisher={arXiv:2409.11404},
url={https://arxiv.org/abs/2409.11404},
}
```

---
license: cc-by-nc-sa-4.0
dataset_info:
- config_name: PIQA-eng
splits:
- name: test
num_examples: 1838
- config_name: PIQA-msa
splits:
- name: test
num_examples: 1838
- config_name: PIQA-lev
splits:
- name: test
num_examples: 1838
- config_name: PIQA-egy
splits:
- name: test
num_examples: 1838
configs:
- config_name: PIQA-eng
data_files:
- split: test
path: PIQA_eng/validation.json
- config_name: PIQA-msa
data_files:
- split: test
path: PIQA_msa/validation.json
- config_name: PIQA-lev
data_files:
- split: test
path: PIQA_lev/validation.json
- config_name: PIQA-egy
data_files:
- split: test
path: PIQA_egy/validation.json
---

# AraDiCE：面向大语言模型方言与文化能力的基准测试集

## 概述
**AraDiCE数据集**旨在评估大语言模型（Large Language Models，LLMs）的方言理解与文化适配能力。该数据集由各类基准数据集的后编辑版本构成，专为与阿拉伯语相关的文化与方言场景下的验证工作精心整理。本仓库展示了该数据集的PIQA划分版本。

<!-- ## 文件/目录结构
待办事项：
- **licenses_by-nc-sa_4.0_legalcode.txt** 许可证说明文件。
- **README.md** 本文档。 -->

## 评估方法
本研究采用[lm-harness](https://github.com/EleutherAI/lm-evaluation-harness)评估框架开展基准测试工作，相关测试结果即将发布，敬请期待！

## 许可协议
本数据集采用**知识共享署名-非商业性使用-相同方式共享4.0国际许可协议（Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License，CC BY-NC-SA 4.0）**进行分发，完整许可文本可参阅附带的`licenses_by-nc-sa_4.0_legalcode.txt`文件。

## 引用信息
请参阅以下论文获取详细信息：<a href="https://arxiv.org/pdf/2409.11404" target="_blank" style="margin-right: 15px; margin-left: 10px">点击此处</a>。

@article{mousi2024aradicebenchmarksdialectalcultural,
title={{AraDiCE}: Benchmarks for Dialectal and Cultural Capabilities in LLMs},
author={Basel Mousi and Nadir Durrani and Fatema Ahmad and Md. Arid Hasan and Maram Hasanain and Tameem Kabbani and Fahim Dalvi and Shammur Absar Chowdhury and Firoj Alam},
year={2024},
publisher={arXiv:2409.11404},
url={https://arxiv.org/abs/2409.11404},
}