Stopword Lists for African Languages

### Context:
Some words, like “the” or “and” in English, are used a lot in speech and writing. For most Natural Language Processing applications, you will want to remove these very frequent words. This is usually done using a list of “stopwords” which has been complied by hand.

### Content:
This project uses the source texts provided by the African Storybook Project as a corpus and provides a number of tools to extract frequency lists and lists of stopwords from this corpus for the 60+ languages covered by ASP.

Included in this dataset are the following languages:

* Afrikaans: stoplist and word frequency
* Hausa: stoplist and word frequency
* Lugbarati: word frequency only
* Lugbarati (Official): word frequency only
* Somali: stoplist and word frequency
* Sesotho: stoplist and word frequency
* Kiswahili: stoplist and word frequency
* Yoruba: stoplist and word frequency
* isiZulu: stoplist and word frequency

Files are named using the language’s ISO code. For each language, code.txt is the list of stopwords, and code_frequency_list.txt is word frequency information. A list of ISO codes the the languages associated with them may be found in ISO_codes.csv.

### Acknowledgements:
This project therefore attempts to fill in the gap in language coverage for African language stoplists by using the freely-available and open-licensed ASP Source project as a corpus.
Dual-licensed under CC-BY and Apache-2.0 license. Compiled by Liam Doherty. More information and the scripts used to generate these files are available [here](https://github.com/dohliam/more-stoplists).

### Inspiration:
This dataset is mainly helpful for use during NLP analysis, however there may some interesting insights in the data.

* What qualities do stopwords share across languages? Given a novel language, could you predict what its stopwords should be?
* What stopwords are shared across languages?
* Often, related languages will have words with the same meaning and similar spellings. Can you automatically identify any of these pairs of words?

### You may also like:

* [Stopword Lists for 19 Languages (mainly European and South Asian)](https://www.kaggle.com/rtatman/stopword-lists-for-19-languages)

{'Context': '某些词汇，例如英语中的“the”或“and”，在口语和书面语中频繁出现。对于大多数自然语言处理应用而言，去除这些高频词汇是必要的。这通常通过手动编纂的“停用词表”来实现。', 'Content': '本项目以非洲故事书项目提供的源文本作为语料库，并提供了若干工具，用以从涵盖非洲故事书项目60多种语言的语料库中提取频率列表和停用词表。该数据集包含以下语言：

* 阿非利堪斯语：停用词表和词频
* 哈乌萨语：停用词表和词频
* 卢加巴提语：仅词频
* 卢加巴提语（官方）：仅词频
* 索马里语：停用词表和词频
* 塞索托语：停用词表和词频
* 斯瓦希里语：停用词表和词频
* 约鲁巴语：停用词表和词频
* 希兹鲁语：停用词表和词频

文件命名采用语言的ISO代码。对于每种语言，code.txt是停用词表，而code_frequency_list.txt是词频信息。ISO代码及其对应语言的列表可查阅ISO_codes.csv文件。', 'Acknowledgements': '因此，本项目旨在填补非洲语言停用词表在语言覆盖面上的空白，通过使用免费且开源的非洲故事书项目源项目作为语料库。本项目采用CC-BY和Apache-2.0双许可协议。由利亚姆·多赫蒂（Liam Doherty）编纂。更多信息和生成这些文件的脚本可在[此处](https://github.com/dohliam/more-stoplists)找到。', 'Inspiration': '此数据集主要适用于自然语言处理分析阶段，然而，数据中可能蕴含一些有趣的洞见。

* 不同语言中的停用词有哪些共同品质？对于一种新型语言，能否预测其停用词的构成？
* 不同语言之间共享的停用词有哪些？
* 相关语言之间往往存在意义相同、拼写相似的词汇。能否自动识别这些词汇对？', 'You may also like': '您可能还会感兴趣：[19种语言的停用词表（主要针对欧洲和南亚）](https://www.kaggle.com/rtatman/stopword-lists-for-19-languages)'}