airesearch/scb_mt_enth_2020

---
annotations_creators:
- crowdsourced
- expert-generated
- found
- machine-generated
language_creators:
- expert-generated
- found
- machine-generated
language:
- en
- th
license:
- cc-by-sa-4.0
multilinguality:
- translation
size_categories:
- 1M<n<10M
source_datasets:
- original
task_categories:
- translation
task_ids: []
paperswithcode_id: scb-mt-en-th-2020
pretty_name: ScbMtEnth2020
dataset_info:
- config_name: enth
features:
- name: translation
dtype:
translation:
languages:
- en
- th
- name: subdataset
dtype: string
splits:
- name: train
num_bytes: 390411946
num_examples: 801402
- name: validation
num_bytes: 54167280
num_examples: 100173
- name: test
num_bytes: 53782790
num_examples: 100177
download_size: 138415559
dataset_size: 498362016
- config_name: then
features:
- name: translation
dtype:
translation:
languages:
- th
- en
- name: subdataset
dtype: string
splits:
- name: train
num_bytes: 390411946
num_examples: 801402
- name: validation
num_bytes: 54167280
num_examples: 100173
- name: test
num_bytes: 53782790
num_examples: 100177
download_size: 138415559
dataset_size: 498362016
---

# Dataset Card for `scb_mt_enth_2020`

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://airesearch.in.th/
- **Repository:** https://github.com/vistec-AI/thai2nmt
- **Paper:** https://arxiv.org/abs/2007.03541
- **Leaderboard:**
- **Point of Contact:** https://airesearch.in.th/

### Dataset Summary

scb-mt-en-th-2020: A Large English-Thai Parallel Corpus
The primary objective of our work is to build a large-scale English-Thai dataset for machine translation.
We construct an English-Thai machine translation dataset with over 1 million segment pairs, curated from various sources,
namely news, Wikipedia articles, SMS messages, task-based dialogs, web-crawled data and government documents.
Methodology for gathering data, building parallel texts and removing noisy sentence pairs are presented in a reproducible manner.
We train machine translation models based on this dataset. Our models' performance are comparable to that of
Google Translation API (as of May 2020) for Thai-English and outperform Google when the Open Parallel Corpus (OPUS) is
included in the training data for both Thai-English and English-Thai translation.
The dataset, pre-trained models, and source code to reproduce our work are available for public use.

### Supported Tasks and Leaderboards

machine translation

### Languages

English, Thai

## Dataset Structure

### Data Instances

```
{'subdataset': 'aqdf', 'translation': {'en': 'FAR LEFT: Indonesian National Police Chief Tito Karnavian, from left, Philippine National Police Chief Ronald Dela Rosa and Royal Malaysian Police Inspector General Khalid Abu Bakar link arms before the Trilateral Security Meeting in Pasay city, southeast of Manila, Philippines, in June 2017. [THE ASSOCIATED PRESS]', 'th': '(ซ้ายสุด) นายติโต คาร์นาเวียน ผู้บัญชาการตํารวจแห่งชาติอินโดนีเซีย (จากซ้าย) นายโรนัลด์ เดลา โรซา ผู้บัญชาการตํารวจแห่งชาติฟิลิปปินส์ และนายคาลิด อาบู บาการ์ ผู้บัญชาการตํารวจแห่งชาติมาเลเซีย ไขว้แขนกันก่อนเริ่มการประชุมความมั่นคงไตรภาคีในเมืองปาเซย์ ซึ่งอยู่ทางตะวันออกเฉียงใต้ของกรุงมะนิลา ประเทศฟิลิปปินส์ ในเดือนมิถุนายน พ.ศ. 2560 ดิแอสโซซิเอทเต็ด เพรส'}}
{'subdataset': 'thai_websites', 'translation': {'en': "*Applicants from certain countries may be required to pay a visa issuance fee after their application is approved. The Department of State's website has more information about visa issuance fees and can help you determine if an issuance fee applies to your nationality.", 'th': 'ประเภทวีซ่า รวมถึงค่าธรรมเนียม และข้อกําหนดในการสัมภาษณ์วีซ่า จะขึ้นอยู่กับชนิดของหนังสือเดินทาง และจุดประสงค์ในการเดินทางของท่าน โปรดดูตารางด้านล่างก่อนการสมัครวีซ่า'}}
{'subdataset': 'nus_sms', 'translation': {'en': 'Yup... Okay. Cya tmr... So long nvr write already... Dunno whether tmr can come up with 500 words', 'th': 'ใช่...ได้ แล้วเจอกันพรุ่งนี้... นานแล้วไม่เคยเขียน... ไม่รู้ว่าพรุ่งนี้จะทําได้ถึง500คําไหมเลย'}}
```

### Data Fields

- `subdataset`: subdataset from which the sentence pair comes from
- `translation`:
- `en`: English sentences (original source)
- `th`: Thai sentences (originally target for translation)

### Data Splits

```
Split ratio (train, valid, test) : (0.8, 0.1, 0.1)
Number of paris (train, valid, test): 801,402 | 100,173 | 100,177

# Train
generated_reviews_yn: 218,637 ( 27.28% )
task_master_1: 185,671 ( 23.17% )
generated_reviews_translator: 105,561 ( 13.17% )
thai_websites: 93,518 ( 11.67% )
paracrawl: 46,802 ( 5.84% )
nus_sms: 34,495 ( 4.30% )
mozilla_common_voice: 2,451 ( 4.05% )
wikipedia: 26,163 ( 3.26% cd)
generated_reviews_crowd: 19,769 ( 2.47% )
assorted_government: 19,712 ( 2.46% )
aqdf: 10,466 ( 1.31% )
msr_paraphrase: 8,157 ( 1.02% )

# Valid
generated_reviews_yn: 30,786 ( 30.73% )
task_master_1: 18,531 ( 18.50% )
generated_reviews_translator: 13,884 ( 13.86% )
thai_websites: 13,381 ( 13.36% )
paracrawl: 6,618 ( 6.61% )
nus_sms: 4,628 ( 4.62% )
wikipedia: 3,796 ( 3.79% )
assorted_government: 2,842 ( 2.83% )
generated_reviews_crowd: 2,409 ( 2.40% )
aqdf: 1,518 ( 1.52% )
msr_paraphrase: 1,107 ( 1.11% )
mozilla_common_voice: 673 ( 0.67% )

# Test
generated_reviews_yn: 30,785 ( 30.73% )
task_master_1: 18,531 ( 18.50% )
generated_reviews_translator: 13,885 ( 13.86% )
thai_websites: 13,381 ( 13.36% )
paracrawl: 6,619 ( 6.61% )
nus_sms: 4,627 ( 4.62% )
wikipedia: 3,797 ( 3.79% )
assorted_government: 2,844 ( 2.83% )
generated_reviews_crowd: 2,409 ( 2.40% )
aqdf: 1,519 ( 1.52% )
msr_paraphrase: 1,107 ( 1.11% )
mozilla_common_voice : 673 ( 0.67% )
```

## Dataset Creation

### Curation Rationale

[AIResearch](https://airesearch.in.th/), funded by [VISTEC](https://www.vistec.ac.th/) and [depa](https://www.depa.or.th/th/home), curated this dataset as part of public NLP infrastructure. The center releases the dataset and baseline models under CC-BY-SA 4.0.

### Source Data

#### Initial Data Collection and Normalization

The sentence pairs are curated from news, Wikipedia articles, SMS messages, task-based dialogs, webcrawled data and government documents. Sentence pairs are generated by:
- Professional translators
- Crowdsourced translators
- Google Translate API and human annotators (accepted or rejected)
- Sentence alignment with [multilingual universal sentence encoder](https://tfhub.dev/google/universal-sentence-encoder-multilingual/3); the author created [CRFCut](https://github.com/vistec-AI/crfcut) to segment Thai sentences to be abel to align with their English counterparts (sentence segmented by [NLTK](https://www.nltk.org/))

For detailed explanation of dataset curation, see https://arxiv.org/pdf/2007.03541.pdf

### Annotations

#### Sources and Annotation process

- generated_reviews_yn: generated by [CTRL](https://arxiv.org/abs/1909.05858), translated to Thai by Google Translate API and annotated as accepted or rejected by human annotators (we do not include rejected sentence pairs)
- task_master_1: [Taskmaster-1](https://research.google/tools/datasets/taskmaster-1/) translated by professional translators hired by [AIResearch](https://airesearch.in.th/)
- generated_reviews_translator: professional translators hired by [AIResearch](https://airesearch.in.th/)
- thai_websites: webcrawling from top 500 websites in Thailand; respective content creators; the authors only did sentence alignment
- paracrawl: replicating Paracrawl's methodology for webcrawling; respective content creators; the authors only did sentence alignment
- nus_sms: [The National University of Singapore SMS Corpus](https://scholarbank.nus.edu.sg/handle/10635/137343) translated by crowdsourced translators hired by [AIResearch](https://airesearch.in.th/)
- wikipedia: Thai Wikipedia; respective content creators; the authors only did sentence alignment
- assorted_government: Government document in PDFs from various government websites; respective content creators; the authors only did sentence alignment
- generated_reviews_crowd: generated by [CTRL](https://arxiv.org/abs/1909.05858), translated to Thai by crowdsourced translators hired by [AIResearch](https://airesearch.in.th/)
- aqdf: Bilingual news from [Asia Pacific Defense Forum](https://ipdefenseforum.com/); respective content creators; the authors only did sentence alignment
- msr_paraphrase: [Microsoft Research Paraphrase Corpus](https://www.microsoft.com/en-us/download/details.aspx?id=52398) translated to Thai by crowdsourced translators hired by [AIResearch](https://airesearch.in.th/)
- mozilla_common_voice: English version of [Mozilla Common Voice](https://commonvoice.mozilla.org/) translated to Thai by crowdsourced translators hired by [AIResearch](https://airesearch.in.th/)

### Personal and Sensitive Information

There are risks of personal information to be included in the webcrawled data namely `paracrawl` and `thai_websites`.

## Considerations for Using the Data

### Social Impact of Dataset

- The first and currently largest English-Thai machine translation dataset that is strictly cleaned and deduplicated, compare to other sources such as Paracrawl.

### Discussion of Biases

- Gender-based ending honorifics in Thai (ครับ/ค่ะ) might not be balanced due to more female translators than male for `task_master_1`

### Other Known Limitations

#### Segment Alignment between Languages With and Without Boundaries
Unlike English, there is no segment boundary marking in Thai. One segment in Thai may or may not cover all
the content of an English segment. Currently, we mitigate this problem by grouping Thai segments together before
computing the text similarity scores. We then choose the combination with the highest text similarity score. It can be
said that adequacy is the main issue in building this dataset.
Quality of Translation from Crawled Websites
Some websites use machine translation models such as Google Translate to localize their content. As a result, Thai
segments retrieved from web crawling might face issues of fluency since we do not use human annotators to perform
quality control.

#### Quality Control of Crowdsourced Translators
When we use a crowdsourcing platform to translate the content, we can not fully control the quality of the translation.
To combat this, we filter out low-quality segments by using a text similarity threshold, based on cosine similarity of
universal sentence encoder vectors. Moreover, some crowdsourced translators might copy and paste source segments to
a translation engine and take the results as answers to the platform. To further improve, we can apply techniques such
as described in [Zaidan, 2012] to control the quality and avoid fraud on the platform.

#### Domain Dependence of Machine Tranlsation Models
We test domain dependence of machine translation models by comparing models trained and tested on the same dataset,
using 80/10/10 train-validation-test split, and models trained on one dataset and tested on the other.

## Additional Information

### Dataset Curators

[AIResearch](https://airesearch.in.th/), funded by [VISTEC](https://www.vistec.ac.th/) and [depa](https://www.depa.or.th/th/home)

### Licensing Information

CC-BY-SA 4.0

### Citation Information

```
@article{lowphansirikul2020scb,
title={scb-mt-en-th-2020: A Large English-Thai Parallel Corpus},
author={Lowphansirikul, Lalita and Polpanumas, Charin and Rutherford, Attapol T and Nutanong, Sarana},
journal={arXiv preprint arXiv:2007.03541},
year={2020}
}
```

### Contributions

Thanks to [@cstorm125](https://github.com/cstorm125) for adding this dataset.

原始信息汇总

数据集概述

数据集基本信息

• 数据集名称: scb-mt-en-th-2020
• 数据集别名: ScbMtEnth2020
• 数据集ID: paperswithcode_id: scb-mt-en-th-2020
• 数据集大小: 1M<n<10M 条数据
• 语言: 英语 (en), 泰语 (th)
• 许可证: CC-BY-SA-4.0
• 多语言性: 翻译 (translation)
• 任务类别: 翻译 (translation)
• 数据来源: 原始数据 (original)

数据集结构

数据实例

数据集包含以下字段:

• subdataset: 数据子集来源
• translation: 翻译内容
  • en: 英语原文
  • th: 泰语翻译

数据分割

数据集分为三个部分:

• 训练集: 801,402 条数据
• 验证集: 100,173 条数据
• 测试集: 100,177 条数据

数据集创建

数据来源

数据集内容来源于以下几个方面:

• 新闻
• 维基百科文章
• SMS 短信
• 任务型对话
• 网络爬取数据
• 政府文件

数据注释

数据注释方式包括:

• 专业翻译
• 众包翻译
• 机器生成翻译
• 人工标注接受或拒绝的翻译结果

个人和敏感信息

数据中可能包含来自网络爬取的个人敏感信息。

使用数据集的考虑

社会影响

该数据集是目前最大的严格清洗和去重的英语-泰语机器翻译数据集。

偏见讨论

数据集可能存在性别偏见，因为翻译者中女性多于男性。

其他已知限制

• 泰语和英语在句子边界上的不匹配问题
• 网络爬取数据的翻译质量问题
• 众包翻译的质量控制问题

附加信息

数据集维护者

• AIResearch, 由 VISTEC 和 depa 资助

许可证信息

• CC-BY-SA 4.0

引用信息

@article{lowphansirikul2020scb, title={scb-mt-en-th-2020: A Large English-Thai Parallel Corpus}, author={Lowphansirikul, Lalita and Polpanumas, Charin and Rutherford, Attapol T and Nutanong, Sarana}, journal={arXiv preprint arXiv:2007.03541}, year={2020} }

搜集汇总

数据集介绍

构建方式

在机器翻译领域，高质量平行语料库的构建是推动模型性能提升的关键基石。scb-mt-en-th-2020数据集由AIResearch团队精心打造，旨在为英语-泰语机器翻译提供大规模、高质量的训练资源。该数据集的构建过程融合了多种创新策略：通过专业翻译人员、众包译者以及基于Google Translate API的人工审核机制生成句子对；利用多语言通用句子编码器进行句子对齐，并借助CRFCut工具对泰语进行分词处理以解决无边界标记语言的挑战。数据来源涵盖新闻、维基百科、短信、任务型对话、网络爬取内容及政府文档，最终经过严格清洗与去重，形成了超过100万对高质量平行句段。

特点

该数据集展现出卓越的多样性与规模优势，其包含超过100万对平行句段，训练集、验证集与测试集按8:1:1比例划分。数据来源极为丰富，涵盖12个子数据集，如来自CTRL生成并由人工标注的评论、Taskmaster-1对话的专业翻译、泰语顶级网站的网络爬取内容、Paracrawl方法复制、新加坡国立大学短信语料库的众包翻译、泰语维基百科、政府文档、亚太防务论坛双语新闻、微软研究院释义语料库以及Mozilla Common Voice语音数据的翻译版本。这种多源融合策略确保了数据集在领域覆盖上的广泛性，同时通过文本相似度阈值过滤与人工质量把控，有效降低了噪声与低质量翻译的影响。

使用方法

研究人员可通过HuggingFace Datasets库便捷加载该数据集，支持两种配置：'enth'（英语-泰语）与'then'（泰语-英语），以适应不同翻译方向的需求。数据集以标准格式存储，包含'subdataset'字段标识来源，以及'translation'字段下的'en'与'th'双语内容。用户可基于801,402条训练样本、100,173条验证样本与100,177条测试样本开展机器翻译模型的训练与评估。此外，数据集采用CC-BY-SA 4.0许可协议，支持学术研究与商业应用，团队还提供了预训练模型与复现代码，便于研究者快速上手与对比实验。

背景与挑战

背景概述

机器翻译作为自然语言处理领域的核心任务，其性能高度依赖于大规模、高质量的平行语料库。然而，对于英语-泰语这一语言对而言，长期以来缺乏公开可用的规模化数据集，严重制约了相关研究的发展。为填补这一空白，泰国人工智能研究机构AIResearch与VISTEC、depa合作，于2020年发布了scb-mt-en-th-2020数据集。该数据集由Lalita Lowphansirikul、Charin Polpanumas等研究者主导构建，包含超过100万个句对，来源涵盖新闻、维基百科、短信、任务型对话、网络爬取数据及政府文档，旨在为英泰机器翻译提供坚实的基础资源。其发布不仅显著推动了低资源语言翻译的研究，还通过开源模型与代码，为学术界和工业界树立了可复现的标杆，影响力辐射至东南亚乃至全球自然语言处理社区。

当前挑战

该数据集所解决的领域挑战在于英泰机器翻译中平行语料匮乏的问题，尤其是泰语作为无分词边界的语言，与英语的句子对齐存在根本性困难。构建过程中，研究团队面临多重挑战：首先，泰语缺乏明确的句子边界标记，导致跨语言片段对齐时需依赖分组计算文本相似度，但可能牺牲翻译的充分性；其次，从网络爬取的泰语文本常由机器翻译生成，存在流畅性不足的问题，而人工翻译质量因众包平台难以完全控制，需要引入句子编码器进行相似度阈值过滤；此外，不同子数据集（如政府文档、社交媒体）的领域差异显著，模型在跨域测试时表现不稳定，凸显了领域依赖性的挑战。这些难题共同构成了数据集构建与使用的核心瓶颈。

常用场景

经典使用场景

在神经机器翻译领域，scb-mt-en-th-2020数据集被广泛用于英-泰平行语料的模型训练与评估。该数据集汇聚了来自新闻、维基百科、短信、任务型对话、网页抓取及政府文档等多源异构的百万级句对，为构建高鲁棒性的翻译系统提供了丰富的语料基础。研究者常将其作为基准数据集，用以训练Transformer等主流序列到序列模型，并通过标准的80/10/10划分进行性能验证，从而推动低资源语言对翻译质量的提升。

实际应用

在实际应用中，该数据集支撑了泰语与英语之间的自动翻译服务，广泛应用于旅游、电商、政府文档本地化及跨语言信息检索等场景。例如，基于该数据集训练的模型可嵌入实时聊天系统中，辅助跨国商务沟通；亦可用于新闻媒体的多语言内容生成，提升信息传播效率。此外，其涵盖的短信、对话等口语化语料，有效增强了翻译系统对非正式表达的适应能力，促进了人机交互的流畅性。

衍生相关工作

该数据集衍生了一系列经典工作，包括但不限于：基于CRFCut的泰语句子分割工具，解决了无边界语言的对齐难题；利用多语言通用句子编码器进行语义相似度过滤，提高了语料质量；以及通过领域迁移实验揭示了翻译模型对训练数据分布的依赖性。此外，研究者还基于该数据集发布了预训练模型与开源代码，推动了泰语自然语言处理社区的协作发展，并启发了后续针对低资源语言的大规模平行语料构建方法论。