Cultural-Evaluation-Kalahi

# Kalahi

Kalahi evaluates the ability of LLMs to generate responses relevant to Filipino culture in terms of shared knowledge and ethics. This dataset contains a MCQ-compatible version of the [Kalahi](https://arxiv.org/abs/2409.15380) dataset that is used in [SEA-HELM](https://leaderboard.sea-lion.ai/).

### Supported Tasks and Leaderboards

Kalahi is designed for evaluating Filipino cultural representations in instruction-tuned large language models (LLMs). It is part of the [SEA-HELM](https://leaderboard.sea-lion.ai/) leaderboard from [AI Singapore](https://aisingapore.org/).

### Languages
- Tagalog (tl)

### Dataset Details
Kalahi only has a Tagalog (tl) split. Below are the statistics for this dataset. The number of tokens only refer to the strings of text found within the `prompts` column.

| Split | # of examples | # of GPT-4o tokens | # of Gemma 2 tokens | # of Llama 3 tokens |
|-|:-|:-|:-|:-|
| tl | 150 | 23710 | 26534 | 29766 |

### Data Sources

| Data Source | License | Language/s | Split/s
|-|:-|:-| :-|
| [Kalahi](https://huggingface.co/datasets/aisingapore/kalahi) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | Tagalog | tl

### License

For the license/s of the dataset/s, please refer to the data sources table above.

We endeavor to ensure data used is permissible and have chosen datasets from creators who have processes to exclude copyrighted or disputed data.

## Acknowledgement

This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA),
Singapore under its National Large Language Model Funding Initiative.

### References

```bibtex
@misc{montalan2024kalahihandcraftedgrassrootscultural,
title={Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino},
author={Jann Railey Montalan and Jian Gang Ngui and Wei Qi Leong and Yosephine Susanto and Hamsawardhini Rengarajan and William Chandra Tjhi and Alham Fikri Aji},
year={2024},
eprint={2409.15380},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2409.15380},
}
```

# Kalahi

Kalahi 用于评估大语言模型（Large Language Model, LLM）生成贴合菲律宾文化共享知识与伦理规范的回复的能力。本数据集包含[Kalahi](https://arxiv.org/abs/2409.15380)数据集的多项选择题（Multiple Choice Question, MCQ）兼容格式版本，该版本已被应用于[SEA-HELM](https://leaderboard.sea-lion.ai/)评测平台。

### 支持任务与评测榜单

Kalahi 旨在评估经过指令微调的大语言模型对菲律宾文化的表征能力，该数据集隶属于[AI Singapore](https://aisingapore.org/)推出的[SEA-HELM](https://leaderboard.sea-lion.ai/)评测榜单。

### 语言

- 他加禄语（Tagalog, tl）

### 数据集详情

Kalahi 仅包含他加禄语（tl）划分集。以下为本数据集的统计信息，Token（Token）数量仅统计`prompts`列中的文本字符串。

| 划分集 | 示例数量 | GPT-4o Token 数量 | Gemma 2 Token 数量 | Llama 3 Token 数量 |
|:-|:-|:-|:-|:-|
| tl | 150 | 23710 | 26534 | 29766 |

### 数据来源

| 数据来源 | 授权协议 | 语言 | 划分集 |
|:-|:-|:-|:-|
| [Kalahi](https://huggingface.co/datasets/aisingapore/kalahi) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | 他加禄语 | tl |

### 授权协议

数据集的授权协议请参阅上文的数据来源表格。

我们致力于确保所用数据合规，并仅选用那些具备排除受版权保护或争议性数据流程的创作者所提供的数据集。

## 致谢

本项目获得新加坡国家研究基金会及新加坡资讯通信媒体发展局（IMDA）旗下国家大语言模型资助计划的支持。

### 参考文献

bibtex
@misc{montalan2024kalahihandcraftedgrassrootscultural,
title={Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino},
author={Jann Railey Montalan and Jian Gang Ngui and Wei Qi Leong and Yosephine Susanto and Hamsawardhini Rengarajan and William Chandra Tjhi and Alham Fikri Aji},
year={2024},
eprint={2409.15380},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2409.15380},
}