thai_exam

# Dataset Card for Thai\_Exam

ThaiExam is a Thai knowledge benchmarking dataset, consisting of multiple-choice questions from examinations in Thailand. The dataset was originally developed for evaluating [Typhoon (Thai LLM)](https://arxiv.org/abs/2312.13951). This dataset contains 5 splits corresponding to 5 examinations as follows:

1. **ONET**: The Ordinary National Educational Test (ONET) is an examination for students in Thailand. This dataset is based on the grade-12 ONET exam, comprising 4 subjects and each question has 5 choices. These subjects are Thai, Mathematics, Social Studies, and Science, amounting to a total of 167 questions and options.

2. **IC**: The Investment Consultant (IC) examination is a licensing test for investment professionals in Thailand. The exam questions were developed by the Stock Exchange of Thailand (SET), featuring 4 choices per question. We extracted questions for levels 1, 2, and 3 resulting in a total of 100 questions and options.

3. **TGAT**: The Thai General Aptitude Test (TGAT) is a national high school examination in Thailand. The exam focuses on critical and logical thinking skills. We collected a total of 70 questions and answers where each question consists of 4 choices per question.

4. **TPAT-1**: The Thai Professional Aptitude Test 1 (TPAT-1) is a national high school examination in Thailand. The Exam assesses students’ professional skills requirement in medical schools. This subset contains reasoning and medical ethics. We collected a total of 121 questions and answers. The TPAT-1 consists of 5 choices per question.

5. **A-Level**: The A-Level exam is an academic knowledge assessment examination (Applied Knowledge Level) covering general foundational subjects. The content assessed in this examination aligns with the curriculum guidelines and emphasizes the practical application of knowledge in daily life. We collected a total of 132 questions and answers.

# Usage
For each exam, the train split, consisting of 5 examples, can be used for 5-shot evaluation (in-context examples), and we provide a 5-shot evaluation script [here]().

# Version
This dataset was initially developed for evaluating Typhoon. As we made improvements and fixed errors in the dataset, here are the versions:
- `v0.1`: As per results reported in [Typhoon technical report](https://arxiv.org/abs/2312.13951) and [Typhoon-1.5 Blog](https://blog.opentyphoon.ai/typhoon-1-5-release-a9364cb8e8d7). This pre-release version can be downloaded from [this link]().
- `v1.0`: The current version (`revision: d78aef04ea3cc5095545e6951cb39e17c64e26a1`)

# HELM
This dataset has been integrated into [the HELM leaderboard](https://crfm.stanford.edu/helm/thai/latest/#leaderboard) by Stanford CRFM.

# Citation Information
This dataset was curated by the Typhoon team, and the first iteration of this dataset was used in the evaluation of Typhoon. If you find this dataset useful, you can cite our work:
```
@article{pipatanakul2023typhoon,
title={Typhoon: Thai Large Language Models},
author={Pipatanakul, Kunat and Jirabovonvisut, Phatrasek and Manakul, Potsawee and Sripaisarnmongkol, Sittipong and Patomwong, Ruangsak and Chokchainant, Pathomporn and Tharnpipitchai, Kasima},
journal={arXiv preprint arXiv:2312.13951},
year={2023}
}
```

# 泰语考试（Thai_Exam）数据集卡片
泰语考试（ThaiExam）是一套泰语知识基准评测数据集，收录源自泰国各类考试的单项选择题。本数据集最初为评测台风（Typhoon，泰语大语言模型）而开发，相关参考链接为[https://arxiv.org/abs/2312.13951]。本数据集包含5个拆分子集，分别对应5类考试，详情如下：
1. **普通全国教育考试（Ordinary National Educational Test，ONET）**：该考试面向泰国学生。本子集基于12年级普通全国教育考试命题，涵盖4门科目，每道题目均设5个选项。涉及科目包括泰语、数学、社会研究与科学，总计167道题目及对应选项。
2. **投资顾问（Investment Consultant，IC）**考试：该考试是泰国投资专业人士的执业资格测试。本考试题目由泰国证券交易所（Stock Exchange of Thailand，SET）命制，每道题目设4个选项。我们提取了第1、2、3级别的考题，总计100道题目及对应选项。
3. **泰国综合能力倾向测试（Thai General Aptitude Test，TGAT）**：泰国全国性高中学力考试，侧重批判性与逻辑思维能力测评。我们共收录70道题目及参考答案，每道题目设4个选项。
4. **泰国专业能力倾向测试1（Thai Professional Aptitude Test 1，TPAT-1）**：泰国全国性高中学力考试，用于评估学生是否具备医学院校要求的专业技能。该子集包含推理题与医学伦理题，我们共收录121道题目及参考答案，每道题目设5个选项。
5. **A-Level考试**：即应用知识水平考试（Applied Knowledge Level），是针对通用基础学科的学术知识测评考试。本次考试的测评内容符合课程大纲要求，侧重知识在日常生活中的实际应用。我们共收录132道题目及参考答案。

# 使用方式
针对每类考试，包含5个样本的训练拆分子集可用于少样本（Few-shot）评测（即上下文示例），我们在此处提供了少样本评测脚本[此处]()。

# 版本信息
本数据集最初为评测台风（Typhoon）而开发。随着我们对数据集进行优化与错误修复，各版本详情如下：
- `v0.1`：对应[台风技术报告](https://arxiv.org/abs/2312.13951)与[台风1.5版本博客](https://blog.opentyphoon.ai/typhoon-1-5-release-a9364cb8e8d7)中公布的测试结果。该预发布版本可通过[此链接]()下载。
- `v1.0`：当前版本（修订号：`d78aef04ea3cc5095545e6951cb39e17c64e26a1`）

# HELM评测集成
本数据集已被斯坦福大学CRFM团队集成至[HELM评测榜单](https://crfm.stanford.edu/helm/thai/latest/#leaderboard)中。

# 引用信息
本数据集由台风（Typhoon）团队整理制作，其首个版本被用于台风模型的评测工作。若您认为本数据集对您的研究有所帮助，可引用如下文献：

@article{pipatanakul2023typhoon,
title={Typhoon: Thai Large Language Models},
author={Pipatanakul, Kunat and Jirabovonvisut, Phatrasek and Manakul, Potsawee and Sripaisarnmongkol, Sittipong and Patomwong, Ruangsak and Chokchainant, Pathomporn and Tharnpipitchai, Kasima},
journal={arXiv preprint arXiv:2312.13951},
year={2023}
}