MultiTurn-Chat-MT-Bench

# SEA-MTBench

SEA-MTBench evaluates a model's ability to engage in multi-turn (2 turns) conversations and respond in ways that align with human needs. We use `gpt-4-1106-preview` as the judge model and compare against `gpt-3.5-turbo-0125` as the baseline model. It is based on [MT-Bench](https://arxiv.org/abs/2306.05685) and was manually translated by native speakers for Indonesian (id), Javanese (jv), Sundanese (su), and Vietnamese (vi). The Thai split of this dataset uses [MT-Bench Thai](https://huggingface.co/datasets/ThaiLLM-Leaderboard/mt-bench-thai) from the [ThaiLLM leaderboard](https://huggingface.co/spaces/ThaiLLM-Leaderboard/leaderboard).

### Supported Tasks and Leaderboards

SEA-MTBench is designed for evaluating chat or instruction-tuned large language models (LLMs). It is part of the [SEA-HELM](https://leaderboard.sea-lion.ai/) leaderboard from [AI Singapore](https://aisingapore.org/).

### Languages
- English (en)
- Indonesian (id)
- Javanese (jv)
- Sundanese (su)
- Thai (th)
- Vietnamese (vi)

### Dataset Details
SEA-MTBench is split by language. Below are the statistics for this dataset. The number of tokens only refer to the strings of text found within the `prompts` column.

| Split | # of examples | # of GPT-4o tokens | # of Gemma 2 tokens | # of Llama 3 tokens |
|-|:-|:-|:-|:-|
| en | 58 | 3606 | 3885 | 3697 |
| id | 58 | 4587 | 4388 | 5631 |
| jv | 58 | 4538 | 5033 | 5740 |
| su | 58 | 4835 | 5591 | 6477 |
| th | 91 | 9660 | 9488 | 1101 |
| vi | 58 | 5167 | 5105 | 4935 |
| **total** | 381 | 32393 | 33490 | 37491 |

### Data Sources

| Data Source | License | Language/s | Split/s
|-|:-|:-| :-|
| [MT-Bench](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | English | en
| [MT-Bench Thai](https://huggingface.co/datasets/ThaiLLM-Leaderboard/mt-bench-thai) | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) | Thai | th
| SEA-MTBench^ | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | Indonesian, Javanese, Sundanese, Vietnamese | id, jv, su, vi

^ manually translated from MT-Bench

### License

For the license/s of the dataset/s, please refer to the data sources table above.

We endeavor to ensure data used is permissible and have chosen datasets from creators who have processes to exclude copyrighted or disputed data.

## Acknowledgement

This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA),
Singapore under its National Large Language Model Funding Initiative.

### References

```bibtex
@misc{zheng2023judgingllmasajudgemtbenchchatbot,
title={Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena},
author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
year={2023},
eprint={2306.05685},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2306.05685},
}
```

# SEA-MTBench

SEA-MTBench 用于评估模型参与多轮（2轮）对话并生成符合人类需求的响应的能力。我们采用`gpt-4-1106-preview`作为评判模型，并以`gpt-3.5-turbo-0125`作为基线模型进行对比。该数据集基于[MT-Bench](https://arxiv.org/abs/2306.05685)，并由母语者手动翻译为印尼语（id）、爪哇语（jv）、巽他语（su）和越南语（vi）。该数据集的泰语分支采用了[ThaiLLM排行榜](https://huggingface.co/spaces/ThaiLLM-Leaderboard/leaderboard)中的[MT-Bench Thai](https://huggingface.co/datasets/ThaiLLM-Leaderboard/mt-bench-thai)。

### 支持任务与排行榜

SEA-MTBench 专为评估聊天模型或指令微调大语言模型（LLM）而设计，隶属于[新加坡AI（AI Singapore）](https://aisingapore.org/)推出的[SEA-HELM](https://leaderboard.sea-lion.ai/)排行榜。

### 支持语言
- 英语（en）
- 印尼语（id）
- 爪哇语（jv）
- 巽他语（su）
- 泰语（th）
- 越南语（vi）

### 数据集详情
SEA-MTBench 按语言划分数据集分支。以下为该数据集的统计信息，其中Token数仅指`prompts`列中的文本字符串。

| 分支 | 样本数量 | GPT-4o Token数 | Gemma 2 Token数 | Llama 3 Token数 |
|-|:-|:-|:-|:-|
| en | 58 | 3606 | 3885 | 3697 |
| id | 58 | 4587 | 4388 | 5631 |
| jv | 58 | 4538 | 5033 | 5740 |
| su | 58 | 4835 | 5591 | 6477 |
| th | 91 | 9660 | 9488 | 1101 |
| vi | 58 | 5167 | 5105 | 4935 |
| **总计** | 381 | 32393 | 33490 | 37491 |

### 数据来源

| 数据来源 | 授权协议 | 支持语言 | 分支 |
|-|:-|:-| :-|
| [MT-Bench](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | 英语 | en
| [MT-Bench Thai](https://huggingface.co/datasets/ThaiLLM-Leaderboard/mt-bench-thai) | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) | 泰语 | th
| SEA-MTBench^ | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | 印尼语、爪哇语、巽他语、越南语 | id, jv, su, vi

^ 由MT-Bench手动翻译而来

### 授权协议

有关数据集的授权协议，请参阅上方的数据来源表格。
我们致力于确保所用数据合规，仅选用那些具备流程以排除受版权保护或存在争议的数据的创作者所提供的数据集。

## 致谢

本项目获得新加坡国家研究基金会及新加坡资讯通信媒体发展局（IMDA）旗下国家大语言模型资助计划的支持。

### 参考文献

bibtex
@misc{zheng2023judgingllmasajudgemtbenchchatbot,
title={Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena},
author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric P. Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
year={2023},
eprint={2306.05685},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2306.05685},
}