Instruction-Following-IFEval

# SEA-IFEval

SEA-IFEval evaluates a model's ability to adhere to constraints provided in the prompt, for example beginning a response with a specific word/phrase or answering with a certain number of sections. It is based on [IFEval](https://arxiv.org/abs/2311.07911) and was manually translated by native speakers for Indonesian, Javanese, Sundanese, Thai, Tagalog, and Vietnamese.

### Supported Tasks and Leaderboards

SEA-IFEval is designed for evaluating chat or instruction-tuned large language models (LLMs). It is part of the [SEA-HELM](https://leaderboard.sea-lion.ai/) leaderboard from [AI Singapore](https://aisingapore.org/).

### Languages
- Indonesian (id)
- Javanese (jv)
- Sundanese (su)
- Tagalog (tl)
- Thai (th)
- Vietnamese (vi)

### Dataset Details
SEA-IFEval is split by language. Below are the statistics for this dataset. The number of tokens only refer to the strings of text found within the `prompts` column.

| Split | # of examples | # of GPT-4o tokens | # of Gemma 2 tokens | # of Llama 3 tokens |
|-|:-|:-|:-|:-|
| en | 105 | 3545 | 3733 | 3688 |
| id | 105 | 4512 | 4146 | 5444 |
| jv | 105 | 4409 | 4901 | 5654 |
| su | 105 | 4762 | 5651 | 6525 |
| th | 105 | 5905 | 5472 | 7035 |
| tl | 105 | 5525 | 5987 | 6736 |
| vi | 105 | 5217 | 5069 | 5171 |
| **total** | 735 | 33875 | 34959 | 40253 |

### Data Sources

| Data Source | License | Language/s | Split/s
|-|:-|:-| :-|
| [IFEval](https://huggingface.co/datasets/google/IFEval) | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) | English | en
| SEA-IFEval^ | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | Indonesian, Javanese, Tagalog, Sundanese, Vietnamese | id, jv, tl, su, vi

^ manually translated from IFEval

### License

For the license/s of the dataset/s, please refer to the data sources table above.

We endeavor to ensure data used is permissible and have chosen datasets from creators who have processes to exclude copyrighted or disputed data.

## Acknowledgement

This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA),
Singapore under its National Large Language Model Funding Initiative.

### References

```bibtex
@misc{zhou2023instructionfollowingevaluationlargelanguage,
title={Instruction-Following Evaluation for Large Language Models},
author={Jeffrey Zhou and Tianjian Lu and Swaroop Mishra and Siddhartha Brahma and Sujoy Basu and Yi Luan and Denny Zhou and Le Hou},
year={2023},
eprint={2311.07911},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2311.07911},
}
```

# SEA-IFEval

SEA-IFEval 用于评估模型遵循提示中给定约束的能力，例如以特定单词/短语作为回复开头，或以指定数量的分段进行回答。该数据集基于[IFEval](https://arxiv.org/abs/2311.07911)构建，并由母语者手动将其翻译为印尼语、爪哇语、巽他语、他加禄语、泰语和越南语。

### 支持任务与排行榜

SEA-IFEval 专为评估聊天型或指令微调大语言模型（Large Language Model，LLM）而设计，它隶属于[AI Singapore](https://aisingapore.org/)推出的[SEA-HELM](https://leaderboard.sea-lion.ai/)排行榜。

### 支持语言
- 印尼语（id）
- 爪哇语（jv）
- 巽他语（su）
- 他加禄语（tl）
- 泰语（th）
- 越南语（vi）

### 数据集详情

SEA-IFEval 按语言划分子集。以下为该数据集的统计信息，其中Token数量仅统计`prompts`列中的文本字符串。

| 拆分子集 | 样本数量 | GPT-4o Token 数 | Gemma 2 Token 数 | Llama 3 Token 数 |
|:-|:-|:-|:-|:-|
| en（英语） | 105 | 3545 | 3733 | 3688 |
| id（印尼语） | 105 | 4512 | 4146 | 5444 |
| jv（爪哇语） | 105 | 4409 | 4901 | 5654 |
| su（巽他语） | 105 | 4762 | 5651 | 6525 |
| th（泰语） | 105 | 5905 | 5472 | 7035 |
| tl（他加禄语） | 105 | 5525 | 5987 | 6736 |
| vi（越南语） | 105 | 5217 | 5069 | 5171 |
| **总计** | 735 | 33875 | 34959 | 40253 |

### 数据来源

| 数据来源 | 授权协议 | 支持语言 | 拆分子集 |
|:-|:-|:-|:-|
| [IFEval](https://huggingface.co/datasets/google/IFEval) | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) | 英语 | en |
| SEA-IFEval^ | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | 印尼语、爪哇语、他加禄语、巽他语、越南语 | id、jv、tl、su、vi |

^ 该数据集由IFEval手动翻译得到。

### 授权协议

有关数据集的授权协议，请参阅上文的数据来源表格。我们致力于确保所用数据合规，所选数据集均来自具备排除受版权保护或存在争议数据流程的创作者。

### 致谢

本项目由新加坡国家研究基金会及新加坡资讯通信媒体发展局（IMDA）依据其国家大语言模型资助计划支持。

### 参考文献

bibtex
@misc{zhou2023instructionfollowingevaluationlargelanguage,
title={Instruction-Following Evaluation for Large Language Models},
author={Jeffrey Zhou and Tianjian Lu and Swaroop Mishra and Siddhartha Brahma and Sujoy Basu and Yi Luan and Denny Zhou and Le Hou},
year={2023},
eprint={2311.07911},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2311.07911},
}