NLU-Metaphor

# SEA Metaphor

SEA Metaphor evaluates a model's ability to interpret paired figurative phrases with divergent meanings. It is sampled from [Multilingual-Fig-QA](https://aclanthology.org/2023.findings-acl.525) for Indonesian, Javanese, and Sundanese.

### Supported Tasks and Leaderboards

SEA Metaphor is designed for evaluating chat or instruction-tuned large language models (LLMs). It is part of the [SEA-HELM](https://leaderboard.sea-lion.ai/) leaderboard from [AI Singapore](https://aisingapore.org/).

### Languages
- Indonesian (id)
- Javanese (jv)
- Sundanese (su)

### Dataset Details
SEA Metaphor is split by language, with additional splits containing fewshot examples. Below are the statistics for this dataset. The number of tokens only refer to the strings of text found within the `prompts` column.

| Split | # of examples | # of GPT-4o tokens | # of Gemma 2 tokens | # of Llama 3 tokens |
|-|:-|:-|:-|:-|
| id | 295 | 10289 | 10120 | 13265
| jv | 282 | 10167 | 10923 | 13279
| su | 295 | 8973 | 10046 | 11858
| id_fewshot | 5 | 183 | 178 | 239
| jv_fewshot | 5 | 154 | 168 | 198
| su_fewshot | 5 | 153 | 165 | 193
| **total** | 887 | 29919 | 31600 | 39032 |

### Data Sources

| Data Source | License | Language/s | Split/s
|-|:-|:-| :-|
| [Multilingual-Fig-QA](https://github.com/simran-khanuja/Multilingual-Fig-QA) | [MIT](https://mit-license.org/) | Indonesian, Javanese, Sundanese | id, id_fewshot, jv, jv_fewshot, su, su_fewshot

### License

For the license/s of the dataset/s, please refer to the data sources table above.

We endeavor to ensure data used is permissible and have chosen datasets from creators who have processes to exclude copyrighted or disputed data.

## Acknowledgement

This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA),
Singapore under its National Large Language Model Funding Initiative.

### References

```bibtex
@inproceedings{kabra-etal-2023-multi,
title = "Multi-lingual and Multi-cultural Figurative Language Understanding",
author = "Kabra, Anubha and
Liu, Emmy and
Khanuja, Simran and
Aji, Alham Fikri and
Winata, Genta and
Cahyawijaya, Samuel and
Aremu, Anuoluwapo and
Ogayo, Perez and
Neubig, Graham",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.findings-acl.525",
doi = "10.18653/v1/2023.findings-acl.525",
pages = "8269--8284",
}
```

# SEA Metaphor
SEA Metaphor 用于评估模型解读语义存在差异的成对比喻性短语的能力。该数据集从面向印尼语、爪哇语与巽他语的 [Multilingual-Fig-QA](https://aclanthology.org/2023.findings-acl.525) 中采样获取。

### 支持任务与评测榜单
SEA Metaphor 专为评估对话式或指令微调大语言模型（Large Language Model, LLM）而设计，隶属于 [AI Singapore](https://aisingapore.org/) 推出的 [SEA-HELM](https://leaderboard.sea-lion.ai/) 评测榜单。

### 支持语言
- 印尼语（id）
- 爪哇语（jv）
- 巽他语（su）

### 数据集详情
SEA Metaphor 按语言划分数据集子集，额外包含带有少样本示例的划分子集。以下为该数据集的统计信息，此处的 Token 数量仅统计 `prompts` 列中的文本字符串。

| 数据集划分 | 示例数量 | GPT-4o Token 数 | Gemma 2 Token 数 | Llama 3 Token 数 |
|:-|:-:|:-:|:-:|:-:|
| id | 295 | 10289 | 10120 | 13265 |
| jv | 282 | 10167 | 10923 | 13279 |
| su | 295 | 8973 | 10046 | 11858 |
| id_fewshot | 5 | 183 | 178 | 239 |
| jv_fewshot | 5 | 154 | 168 | 198 |
| su_fewshot | 5 | 153 | 165 | 193 |
| **总计** | 887 | 29919 | 31600 | 39032 |

### 数据来源
| 数据来源 | 许可证 | 支持语言 | 对应数据集划分 |
|:-|:-|:-|:-|
| [Multilingual-Fig-QA](https://github.com/simran-khanuja/Multilingual-Fig-QA) | [MIT许可证](https://mit-license.org/) | 印尼语、爪哇语、巽他语 | id, id_fewshot, jv, jv_fewshot, su, su_fewshot |

### 许可证
本数据集的许可证信息请参阅上文的数据来源表格。
我们致力于确保所用数据符合合规要求，所选数据集均来自那些具备剔除受版权保护或存在争议数据流程的创作者。

## 致谢
本项目得到新加坡国家研究基金会及新加坡资讯通信媒体发展局（IMDA）旗下国家大语言模型资助计划的支持。

### 参考文献
bibtex
@inproceedings{kabra-etal-2023-multi,
title = "Multi-lingual and Multi-cultural Figurative Language Understanding",
author = "Kabra, Anubha and
Liu, Emmy and
Khanuja, Simran and
Aji, Alham Fikri and
Winata, Genta and
Cahyawijaya, Samuel and
Aremu, Anuoluwapo and
Ogayo, Perez and
Neubig, Graham",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.findings-acl.525",
doi = "10.18653/v1/2023.findings-acl.525",
pages = "8269--8284",
}