uestc-swahili/swahili

---
annotations_creators:
- no-annotation
language_creators:
- expert-generated
language:
- sw
license:
- cc-by-4.0
multilinguality:
- monolingual
size_categories:
- 10K<n<100K
source_datasets:
- original
task_categories:
- text-generation
- fill-mask
task_ids:
- language-modeling
- masked-language-modeling
paperswithcode_id: null
pretty_name: swahili
dataset_info:
features:
- name: text
dtype: string
config_name: swahili
splits:
- name: train
num_bytes: 7700136
num_examples: 42069
- name: test
num_bytes: 695092
num_examples: 3371
- name: validation
num_bytes: 663520
num_examples: 3372
download_size: 2783330
dataset_size: 9058748
---

# Dataset Card for [Dataset Name]

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7339006/
- **Repository:**
- **Paper:** https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7339006/
- **Leaderboard:** [More Information Needed]
- **Point of Contact:** [More Information Needed]

### Dataset Summary

The Swahili dataset developed specifically for language modeling task.
The dataset contains 28,000 unique words with 6.84M, 970k, and 2M words for the train,
valid and test partitions respectively which represent the ratio 80:10:10.
The entire dataset is lowercased, has no punctuation marks and,
the start and end of sentence markers have been incorporated to facilitate easy tokenization during language modeling.

### Supported Tasks and Leaderboards

Language Modeling

### Languages

Swahili (sw)

## Dataset Structure

### Data Instances

[More Information Needed]

### Data Fields

- text : A line of text in Swahili

### Data Splits

train = 80%, valid = 10%, test = 10%

## Dataset Creation

### Curation Rationale

Enhancing African low-resource languages

### Source Data

#### Initial Data Collection and Normalization

The dataset contains 28,000 unique words with 6.84 M, 970k, and 2 M words for the train, valid and test partitions respectively which represent the ratio 80:10:10.
The entire dataset is lowercased, has no punctuation marks and, the start and end of sentence markers have been incorporated to facilitate easy tokenization during language modelling.

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

Unannotated data

#### Who are the annotators?

NA

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

Enhancing African low-resource languages

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

Creative Commons Attribution 4.0 International

### Citation Information

"""\
@InProceedings{huggingface:dataset,
title = Language modeling data for Swahili (Version 1),
authors={Shivachi Casper Shikali, & Mokhosi Refuoe.
},
year={2019},
link = http://doi.org/10.5281/zenodo.3553423
}
"""

### Contributions

Thanks to [@akshayb7](https://github.com/akshayb7) for adding this dataset.

原始信息汇总

数据集卡片

数据集描述

数据集摘要

Swahili数据集专门为语言建模任务开发。数据集包含28,000个唯一单词，训练、验证和测试分区分别包含6.84M、970k和2M单词，比例为80:10:10。整个数据集已小写处理，不含标点符号，并加入了句子开始和结束标记，以便于语言建模时的分词。

支持的任务和排行榜

• 语言建模

语言

Swahili (sw)

数据集结构

数据字段

• text : Swahili中的一行文本

数据分割

• 训练集：80%
• 验证集：10%
• 测试集：10%

数据集创建

策划理由

增强非洲低资源语言

源数据

初始数据收集和规范化

数据集包含28,000个唯一单词，训练、验证和测试分区分别包含6.84M、970k和2M单词，比例为80:10:10。整个数据集已小写处理，不含标点符号，并加入了句子开始和结束标记，以便于语言建模时的分词。

注释

注释过程

未注释数据

注释者

NA

使用数据的注意事项

数据集的社会影响

增强非洲低资源语言

其他信息

许可证信息

Creative Commons Attribution 4.0 International

引用信息

@InProceedings{huggingface:dataset, title = Language modeling data for Swahili (Version 1), authors={Shivachi Casper Shikali, & Mokhosi Refuoe.}, year={2019}, link = http://doi.org/10.5281/zenodo.3553423 }

贡献

感谢@akshayb7添加此数据集。