BillGeate/lst20

---
annotations_creators:
- expert-generated
language_creators:
- found
language:
- th
license:
- other
multilinguality:
- monolingual
size_categories:
- 10K<n<100K
source_datasets:
- original
task_categories:
- token-classification
task_ids:
- named-entity-recognition
- part-of-speech
pretty_name: LST20
tags:
- word-segmentation
- clause-segmentation
- sentence-segmentation
dataset_info:
features:
- name: id
dtype: string
- name: fname
dtype: string
- name: tokens
sequence: string
- name: pos_tags
sequence:
class_label:
names:
'0': NN
'1': VV
'2': PU
'3': CC
'4': PS
'5': AX
'6': AV
'7': FX
'8': NU
'9': AJ
'10': CL
'11': PR
'12': NG
'13': PA
'14': XX
'15': IJ
- name: ner_tags
sequence:
class_label:
names:
'0': O
'1': B_BRN
'2': B_DES
'3': B_DTM
'4': B_LOC
'5': B_MEA
'6': B_NUM
'7': B_ORG
'8': B_PER
'9': B_TRM
'10': B_TTL
'11': I_BRN
'12': I_DES
'13': I_DTM
'14': I_LOC
'15': I_MEA
'16': I_NUM
'17': I_ORG
'18': I_PER
'19': I_TRM
'20': I_TTL
'21': E_BRN
'22': E_DES
'23': E_DTM
'24': E_LOC
'25': E_MEA
'26': E_NUM
'27': E_ORG
'28': E_PER
'29': E_TRM
'30': E_TTL
- name: clause_tags
sequence:
class_label:
names:
'0': O
'1': B_CLS
'2': I_CLS
'3': E_CLS
config_name: lst20
splits:
- name: train
num_bytes: 107725145
num_examples: 63310
- name: validation
num_bytes: 9646167
num_examples: 5620
- name: test
num_bytes: 8217425
num_examples: 5250
download_size: 0
dataset_size: 125588737
---

# Dataset Card for LST20

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://aiforthai.in.th/
- **Repository:**
- **Paper:**
- **Leaderboard:**
- **Point of Contact:** [email]([e-mail])

### Dataset Summary

LST20 Corpus is a dataset for Thai language processing developed by National Electronics and Computer Technology Center (NECTEC), Thailand.
It offers five layers of linguistic annotation: word boundaries, POS tagging, named entities, clause boundaries, and sentence boundaries.
At a large scale, it consists of 3,164,002 words, 288,020 named entities, 248,181 clauses, and 74,180 sentences, while it is annotated with
16 distinct POS tags. All 3,745 documents are also annotated with one of 15 news genres. Regarding its sheer size, this dataset is
considered large enough for developing joint neural models for NLP.
Manually download at https://aiforthai.in.th/corpus.php
See `LST20 Annotation Guideline.pdf` and `LST20 Brief Specification.pdf` within the downloaded `AIFORTHAI-LST20Corpus.tar.gz` for more details.

### Supported Tasks and Leaderboards

- POS tagging
- NER tagging
- clause segmentation
- sentence segmentation
- word tokenization

### Languages

Thai

## Dataset Structure

### Data Instances

```
{'clause_tags': [1, 2, 2, 2, 2, 2, 2, 2, 3], 'fname': 'T11964.txt', 'id': '0', 'ner_tags': [8, 0, 0, 0, 0, 0, 0, 0, 25], 'pos_tags': [0, 0, 0, 1, 0, 8, 8, 8, 0], 'tokens': ['ธรรมนูญ', 'แชมป์', 'สิงห์คลาสสิก', 'กวาด', 'รางวัล', 'แสน', 'สี่', 'หมื่น', 'บาท']}
{'clause_tags': [1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3], 'fname': 'T11964.txt', 'id': '1', 'ner_tags': [8, 18, 28, 0, 0, 0, 0, 6, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 15, 25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 6], 'pos_tags': [0, 2, 0, 2, 1, 1, 2, 8, 2, 10, 2, 8, 2, 1, 0, 1, 0, 4, 7, 1, 0, 2, 8, 2, 10, 1, 10, 4, 2, 8, 2, 4, 0, 4, 0, 2, 8, 2, 10, 2, 8], 'tokens': ['ธรรมนูญ', '_', 'ศรีโรจน์', '_', 'เก็บ', 'เพิ่ม', '_', '4', '_', 'อันเดอร์พาร์', '_', '68', '_', 'เข้า', 'ป้าย', 'รับ', 'แชมป์', 'ใน', 'การ', 'เล่น', 'อาชีพ', '_', '19', '_', 'ปี', 'เป็น', 'ครั้ง', 'ที่', '_', '8', '_', 'ใน', 'ชีวิต', 'ด้วย', 'สกอร์', '_', '18', '_', 'อันเดอร์พาร์', '_', '270']}
```

### Data Fields

- `id`: nth sentence in each set, starting at 0
- `fname`: text file from which the sentence comes from
- `tokens`: word tokens
- `pos_tags`: POS tags
- `ner_tags`: NER tags
- `clause_tags`: clause tags

### Data Splits

| | train | eval | test | all |
|----------------------|-----------|-------------|-------------|-----------|
| words | 2,714,848 | 240,891 | 207,295 | 3,163,034 |
| named entities | 246,529 | 23,176 | 18,315 | 288,020 |
| clauses | 214,645 | 17,486 | 16,050 | 246,181 |
| sentences | 63,310 | 5,620 | 5,250 | 74,180 |
| distinct words | 42,091 | (oov) 2,595 | (oov) 2,006 | 46,692 |
| breaking spaces※ | 63,310 | 5,620 | 5,250 | 74,180 |
| non-breaking spaces※※| 402,380 | 39,920 | 32,204 | 475,504 |

※ Breaking space = space that is used as a sentence boundary marker
※※ Non-breaking space = space that is not used as a sentence boundary marker

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

Respective authors of the news articles

### Annotations

#### Annotation process

Detailed annotation guideline can be found in `LST20 Annotation Guideline.pdf`.

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

All texts are from public news. No personal and sensitive information is expected to be included.

## Considerations for Using the Data

### Social Impact of Dataset

- Large-scale Thai NER & POS tagging, clause & sentence segmentatation, word tokenization

### Discussion of Biases

- All 3,745 texts are from news domain:
- politics: 841
- crime and accident: 592
- economics: 512
- entertainment: 472
- sports: 402
- international: 279
- science, technology and education: 216
- health: 92
- general: 75
- royal: 54
- disaster: 52
- development: 45
- environment: 40
- culture: 40
- weather forecast: 33
- Word tokenization is done accoding to Inter­BEST 2009 Guideline.

### Other Known Limitations

- Some NER tags do not correspond with given labels (`B`, `I`, and so on)

## Additional Information

### Dataset Curators

[NECTEC](https://www.nectec.or.th/en/)

### Licensing Information

1. Non-commercial use, research, and open source

Any non-commercial use of the dataset for research and open-sourced projects is encouraged and free of charge. Please cite our technical report for reference.

If you want to perpetuate your models trained on our dataset and share them to the research community in Thailand, please send your models, code, and APIs to the AI for Thai Project. Please contact Dr. Thepchai Supnithi via [e-mail] for more information.

Note that modification and redistribution of the dataset by any means are strictly prohibited unless authorized by the corpus authors.

2. Commercial use

In any commercial use of the dataset, there are two options.

- Option 1 (in kind): Contributing a dataset of 50,000 words completely annotated with our annotation scheme within 1 year. Your data will also be shared and recognized as a dataset co-creator in the research community in Thailand.

- Option 2 (in cash): Purchasing a lifetime license for the entire dataset is required. The purchased rights of use cover only this dataset.

In both options, please contact Dr. Thepchai Supnithi via [e-mail] for more information.

### Citation Information

```
@article{boonkwan2020annotation,
title={The Annotation Guideline of LST20 Corpus},
author={Boonkwan, Prachya and Luantangsrisuk, Vorapon and Phaholphinyo, Sitthaa and Kriengket, Kanyanat and Leenoi, Dhanon and Phrombut, Charun and Boriboon, Monthika and Kosawat, Krit and Supnithi, Thepchai},
journal={arXiv preprint arXiv:2008.05055},
year={2020}
}
```

### Contributions

Thanks to [@cstorm125](https://github.com/cstorm125) for adding this dataset.

annotations_creators:
- 专家生成（expert-generated）
language_creators:
- 采集得到（found）
language:
- 泰语（th）
license:
- 其他（other）
multilinguality:
- 单语言（monolingual）
size_categories:
- 10K<n<100K
source_datasets:
- 原创数据集（original）
task_categories:
- Token分类（token-classification）
task_ids:
- 命名实体识别（named-entity-recognition）
- 词性标注（part-of-speech）
pretty_name: LST20
tags:
- 词分词（word-segmentation）
- 分句（clause-segmentation）
- 断句（sentence-segmentation）
dataset_info:
features:
- name: id
dtype: 字符串（string）
- name: fname
dtype: 字符串（string）
- name: tokens
sequence: 字符串序列（string）
- name: pos_tags
sequence:
class_label:
names:
'0': NN
'1': VV
'2': PU
'3': CC
'4': PS
'5': AX
'6': AV
'7': FX
'8': NU
'9': AJ
'10': CL
'11': PR
'12': NG
'13': PA
'14': XX
'15': IJ
- name: ner_tags
sequence:
class_label:
names:
'0': O
'1': B_BRN
'2': B_DES
'3': B_DTM
'4': B_LOC
'5': B_MEA
'6': B_NUM
'7': B_ORG
'8': B_PER
'9': B_TRM
'10': B_TTL
'11': I_BRN
'12': I_DES
'13': I_DTM
'14': I_LOC
'15': I_MEA
'16': I_NUM
'17': I_ORG
'18': I_PER
'19': I_TRM
'20': I_TTL
'21': E_BRN
'22': E_DES
'23': E_DTM
'24': E_LOC
'25': E_MEA
'26': E_NUM
'27': E_ORG
'28': E_PER
'29': E_TRM
'30': E_TTL
- name: clause_tags
sequence:
class_label:
names:
'0': O
'1': B_CLS
'2': I_CLS
'3': E_CLS
config_name: lst20
splits:
- name: train
num_bytes: 107725145
num_examples: 63310
- name: validation
num_bytes: 9646167
num_examples: 5620
- name: test
num_bytes: 8217425
num_examples: 5250
download_size: 0
dataset_size: 125588737

# LST20 数据集卡片

## 目录
- [数据集描述](#dataset-description)
- [数据集概述](#dataset-summary)
- [支持任务与基准测试榜单](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据样例](#data-instances)
- [数据字段](#data-fields)
- [数据划分](#data-splits)
- [数据集创建](#dataset-creation)
- [构建初衷](#curation-rationale)
- [源数据](#source-data)
- [标注信息](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集的社会影响](#social-impact-of-dataset)
- [偏差讨论](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集维护者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献致谢](#contributions)

## 数据集描述

- **主页：** https://aiforthai.in.th/
- **代码仓库：**
- **相关论文：**
- **基准测试榜单：**
- **联系人：** [邮箱]([e-mail])

### 数据集概述
LST20语料库是由泰国国家电子与计算机技术中心（National Electronics and Computer Technology Center, NECTEC）开发的泰语自然语言处理数据集。该数据集提供五层语言标注：词边界、词性标注（POS tagging）、命名实体识别（NER）、分句边界与断句边界。其规模可观，总计包含3,164,002个词、288,020个命名实体、248,181个分句以及74,180个句子，同时采用16种不同的词性标注集。全部3,745份文档均标注了15种新闻体裁之一。凭借其体量优势，该数据集足以支撑开发面向自然语言处理的联合神经模型。
可手动下载于 https://aiforthai.in.th/corpus.php，如需更多细节，请查看下载包`AIFORTHAI-LST20Corpus.tar.gz`内的`LST20 Annotation Guideline.pdf`与`LST20 Brief Specification.pdf`。

### 支持任务与基准测试榜单
- 词性标注
- 命名实体识别标注
- 分句
- 断句
- 词分词

### 语言
泰语

## 数据集结构

### 数据样例

{'clause_tags': [1, 2, 2, 2, 2, 2, 2, 2, 3], 'fname': 'T11964.txt', 'id': '0', 'ner_tags': [8, 0, 0, 0, 0, 0, 0, 0, 25], 'pos_tags': [0, 0, 0, 1, 0, 8, 8, 8, 0], 'tokens': ['ธรรมนูญ', 'แชมป์', 'สิงห์คลาสสิก', 'กวาด', 'รางวัล', 'แสน', 'สี่', 'หมื่น', 'บาท']}
{'clause_tags': [1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3], 'fname': 'T11964.txt', 'id': '1', 'ner_tags': [8, 18, 28, 0, 0, 0, 0, 6, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 15, 25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 6], 'pos_tags': [0, 2, 0, 2, 1, 1, 2, 8, 2, 10, 2, 8, 2, 1, 0, 1, 0, 4, 7, 1, 0, 2, 8, 2, 10, 1, 10, 4, 2, 8, 2, 4, 0, 4, 0, 2, 8, 2, 10, 2, 8], 'tokens': ['ธรรมนูญ', '_', 'ศรีโรจน์', '_', 'เก็บ', 'เพิ่ม', '_', '4', '_', 'อันเดอร์พาร์', '_', '68', '_', 'เข้า', 'ป้าย', 'รับ', 'แชมป์', 'ใน', 'การ', 'เล่น', 'อาชีพ', '_', '19', '_', 'ปี', 'เป็น', 'ครั้ง', 'ที่', '_', '8', '_', 'ใน', 'ชีวิต', 'ด้วย', 'สกอร์', '_', '18', '_', 'อันเดอร์พาร์', '_', '270']}

### 数据字段
- `id`：每个集合内的句子序号，从0开始计数
- `fname`：该句子所属的文本文件名称
- `tokens`：词Token（Token）序列
- `pos_tags`：词性标注标签
- `ner_tags`：命名实体识别标注标签
- `clause_tags`：分句标注标签

### 数据划分
| | 训练集 | 验证集 | 测试集 | 总计 |
|----------------------|-----------|-------------|-------------|-----------|
| 词数 | 2,714,848 | 240,891 | 207,295 | 3,163,034 |
| 命名实体数 | 246,529 | 23,176 | 18,315 | 288,020 |
| 分句数 | 214,645 | 17,486 | 16,050 | 246,181 |
| 句子数 | 63,310 | 5,620 | 5,250 | 74,180 |
| 不同词数 | 42,091 | (OOV) 2,595 | (OOV) 2,006 | 46,692 |
| 断句空格※ | 63,310 | 5,620 | 5,250 | 74,180 |
| 非断句空格※※| 402,380 | 39,920 | 32,204 | 475,504 |

※ 断句空格 = 用作句子边界标记的空格
※※ 非断句空格 = 不作为句子边界标记的空格

## 数据集创建

### 构建初衷
[需补充更多信息]

### 源数据
#### 初始数据采集与标准化
[需补充更多信息]

#### 源语言内容创作者是谁？
新闻文章的原作者

### 标注信息
#### 标注流程
详细的标注指南可参见`LST20 Annotation Guideline.pdf`。

#### 标注人员是谁？
[需补充更多信息]

### 个人与敏感信息
所有文本均来自公开新闻，预计不包含任何个人或敏感信息。

## 数据集使用注意事项

### 数据集的社会影响
- 大规模泰语命名实体识别与词性标注、分句与断句、词分词

### 偏差讨论
- 全部3,745份文本均来自新闻领域：
- 政治：841篇
- 犯罪与事故：592篇
- 经济：512篇
- 娱乐：472篇
- 体育：402篇
- 国际：279篇
- 科技与教育：216篇
- 医疗健康：92篇
- 综合：75篇
- 王室：54篇
- 灾害：52篇
- 发展：45篇
- 环境：40篇
- 文化：40篇
- 天气预报：33篇
- 词分词遵循InterBEST 2009指南。

### 其他已知局限性
- 部分命名实体识别标注与给定的（B、I等）标签不匹配

## 附加信息

### 数据集维护者
[NECTEC](https://www.nectec.or.th/en/)

### 许可信息
1. 非商业使用、研究与开源项目
鼓励将本数据集用于非商业性质的研究与开源项目，且无需支付费用。请引用我们的技术报告作为参考文献。
若您希望将基于本数据集训练的模型进行持久化，并分享给泰国国内的研究社区，请将您的模型、代码与API提交至泰国AI for Thai项目。如需更多信息，请联系Thepchai Supnithi博士，邮箱为[e-mail]。
请注意：未经语料库作者授权，严禁以任何方式修改或重新分发本数据集。

2. 商业使用
若需商业使用本数据集，有两种可选方案：
- 方案1（实物对价）：在1年内按照本数据集的标注规范，提交一份包含50,000个已标注词的数据集。您的数据将被共享，并作为联合创作方在泰国研究社区中获得认可。
- 方案2（现金对价）：需购买本完整数据集的终身使用许可。购买的使用权仅覆盖本数据集。
两种方案均需联系Thepchai Supnithi博士，邮箱为[e-mail]以获取更多信息。

### 引用信息

@article{boonkwan2020annotation,
title={The Annotation Guideline of LST20 Corpus},
author={Boonkwan, Prachya and Luantangsrisuk, Vorapon and Phaholphinyo, Sitthaa and Kriengket, Kanyanat and Leenoi, Dhanon and Phrombut, Charun and Boriboon, Monthika and Kosawat, Krit and Supnithi, Thepchai},
journal={arXiv preprint arXiv:2008.05055},
year={2020}
}

### 贡献致谢
感谢 [@cstorm125](https://github.com/cstorm125) 添加本数据集。