mbazaNLP/kinyarwanda_monolingual_v01.0

---
task_categories:
- text-generation
language:
- rw
size_categories:
- 1K<n<10K
---

# <span style="color: red;"> !!! PLEASE USE mbazaNLP/kinyarwanda_monolingual_v01.1 !!! </span>
# <span style="color: red;"> !!! This version contains several duplicates and few non-kinyarwanda documents </span>

### Dataset Summary

The Kinyarwanda Monolingual Dataset version 1 is a large collection of Kinyarwanda language texts aimed at supporting the development of NLP and AI applications which can process Kinyarwanda texts. This dataset contains 78k documents, totalling about 25 million words, and includes diverse content types such as news articles, government reports, religious texts, legal documents, educational materials, and cultural narratives. The dataset has been collected to address the lack of open-source language resources for African languages, with a specific focus on Kinyarwanda.

### Dataset Details

- **Languages**: Kinyarwanda
- **Size**: about 78,000 documents totalling 25 million words (for comparison : Kinyarwanda Bible has about 575k words, so the dataset is equivalent to over 40 bibles)
- **Collection Methodology**: The dataset was created using content available in HTML and PDF formats, covering diverse topics and content types. HTML sources included news sites (e.g., Kigali Today, Igihe), religious sites, Wikipedia, and cultural storytelling websites (e.g., Imigani.rw). PDF documents consisted of government reports, legal texts, transcripts of Senate debates, and educational materials, ensuring a wide representation of language use in different contexts.
- **Content**: The dataset represents a broad cross-section of Kinyarwanda language use, encompassing both formal and informal registers. The content spans news media, cultural stories, bureaucratic documents, legal records, religious teachings, and educational texts.

### Motivation and Background

This dataset aims to address the scarcity of language resources for African languages, particularly in the context of large language models (LLMs). Despite the significant advancements in generative AI, African languages have largely been excluded from these models, limiting their applicability in developing countries. By creating this dataset, we hope to bridge the gap and ensure that the benefits of AI technology can be equitably distributed. The Kinyarwanda Monolingual Dataset also supports Rwanda's national AI policy, which emphasizes data as a critical resource for AI development.

### Limitation and Future Work

This is, to our knowledge, the largest open monolingual dataset in Kinyarwanda. However, its size remains limited. We plan to expand the dataset by identifying additional content sources and addressing the current gaps in representation. We will also continuously improve the data quality, for example by improving text extraction tools.

**Licensing. Use Cases and Usage Limitations**

This dataset is made available under a Creative Commons Attribution 4.0 International License (CC BY 4.0).

The dataset is suitable for fine-tuning existing LLMs to better support African languages, addressing the current gap in language model performance for low-resource languages. It can also be used for other NLP tasks which require a large collection of monolingual texts in Kinyarwanda.

The copyright of the individual texts in this collection remains with their original authors. Reproducing substantial portions of this dataset for presentation to human readers (e.g., in websites, publications, or documents) may conflict with the interests of the copyright holders. Therefore, the use of this dataset should be restricted to data processing with computers for statistical analysis and information extraction, primarily aimed at training AI models to process Kinyarwanda texts—similar to how written materials are used for language learning.

### Citation

Mbaza NLP Community (2024). Kinyarwanda Monolingual Dataset. Version 1.0. <https://huggingface.co/datasets/mbazaNLP/kinyarwanda_monolingual_v01.0>

```
dataset_info:
features:
- name: text
dtype: string
- name: nwords ## number of words
dtype: int64
- name: ntokens_llama32 ## number of tokens LLama 3.2 tokenizers
dtype: int64
splits:
- name: train
num_bytes: 191210626.0
num_examples: 78733
download_size: 112697917
dataset_size: 191210626.0
```

task_categories:
- 文本生成
language:
- 卢旺达语（Kinyarwanda）
size_categories:
- 样本数介于1000至10000之间
---
!!! 请使用 mbazaNLP/kinyarwanda_monolingual_v01.1 版本 !!!
!!! 该版本包含部分重复内容及少量非卢旺达语文本 !!!

### 数据集摘要
卢旺达语单语数据集（Kinyarwanda Monolingual Dataset）版本1是一个大规模卢旺达语文本集合，旨在支持可处理卢旺达语文本的自然语言处理（Natural Language Processing, NLP）与人工智能（Artificial Intelligence, AI）应用开发。本数据集包含7.8万份文档，总字数约2500万，涵盖新闻报道、政府报告、宗教文本、法律文件、教育材料及文化叙事等多样内容类型。本数据集的构建旨在解决非洲语言开源语言资源匮乏的问题，其中重点聚焦卢旺达语。

### 数据集详情
- **语言**：卢旺达语
- **规模**：约7.8万份文档，总字数2500万（对比参考：卢旺达语圣经总字数约57.5万，本数据集规模相当于40部以上圣经）
- **采集方法**：本数据集通过HTML与PDF格式的公开内容构建，涵盖多样主题与内容类型。HTML来源包括新闻网站（如Kigali Today、Igihe）、宗教网站、维基百科及文化故事网站（如Imigani.rw）；PDF文档则包含政府报告、法律文本、参议院辩论记录及教育材料，确保覆盖不同场景下的语言使用场景。
- **内容**：本数据集全面展现了卢旺达语的使用生态，涵盖正式与非正式语体。内容涉及新闻媒体、文化故事、行政公文、法律档案、宗教教义及教育文本。

### 动机与背景
本数据集旨在解决非洲语言（尤其是针对大语言模型（Large Language Model, LLM）场景下）的语言资源稀缺问题。尽管生成式AI（Generative AI）技术取得了显著进展，但非洲语言在这类模型中基本处于缺位状态，限制了其在发展中国家的应用潜力。通过构建本数据集，我们期望填补这一空白，确保AI技术的收益能够公平分配。此外，卢旺达语单语数据集还可支持卢旺达国家AI政策，该政策将数据列为AI发展的关键资源。

### 局限与未来工作
据我们所知，本数据集是目前规模最大的卢旺达语开源单语数据集，但其规模仍有限。我们计划通过挖掘更多内容来源、填补当前的内容覆盖缺口来扩展数据集规模。同时，我们也将持续优化数据质量，例如改进文本提取工具。

**许可协议、使用场景与使用限制**
本数据集采用知识共享署名4.0国际许可协议（Creative Commons Attribution 4.0 International License, CC BY 4.0）发布。
本数据集适用于对现有大语言模型进行微调，以更好地支持非洲语言，填补低资源语言的语言模型性能缺口。此外，它还可用于其他需要大规模卢旺达语单语文本的自然语言处理任务。
本集合中各独立文本的版权仍归原作者所有。若将本数据集的大量内容复现并呈现给人类读者（如用于网站、出版物或文档），可能会侵犯版权持有者的权益。因此，本数据集的使用应仅限于计算机数据处理场景，用于统计分析与信息提取，主要目标是训练可处理卢旺达语文本的AI模型——这与学习语言时使用书面材料的方式类似。

### 引用
姆巴扎NLP社区（Mbaza NLP Community）(2024). 卢旺达语单语数据集. 版本1.0. <https://huggingface.co/datasets/mbazaNLP/kinyarwanda_monolingual_v01.0>

数据集信息：
特征：
- 字段名：text
数据类型：字符串
- 字段名：nwords ## 单词数
数据类型：64位整数
- 字段名：ntokens_llama32 ## Llama 3.2 分词器生成的Token数
数据类型：64位整数
划分：
- 划分集：train
字节数：191210626.0
样本数：78733
下载大小：112697917
数据集总大小：191210626.0