The South African Gov-ZA multilingual corpus

<pre>The South African Gov-ZA multilingual corpus ============================== Github: https://github.com/dsfsi/gov-za-multilingual Zenodo: About Dataset --------------------- The data set contains cabinet statements from the South African government. Data was scraped from the governments website: https://www.gov.za/cabinet-statements The datasets contain government cabinet statements in 11 languages, namely: | Language | Code | Language | Code | |------------|------|------------|------| | English | (eng) | Sepedi | (nso) | | Afrikaans | (afr) | Setswana | (tsn) | | isiNdebele | (nbl) | Siswati | (ssw) | | isiXhosa | (xho) | Tshivenda | (ven) | | isiZulu | (zul) | Xitstonga | (tso) | | Sesotho | (sot) | The dataset contains the full data in a JSON file (/data/govza-cabinet-statements.json), as well as CSV’s split by each language, eg: “govza-cabinet-statements-en.csv” for english. The dataset does not contain special characters like unicode or ascii. Please see the [data-statement.md](/data_statement.md) for full dataset information. *(TODO)* Number of Aligned Pairs with Cosine Similarity Score &gt;= 0.65 ------------------------------------------------------------ | src_lang | trg_lang | num_aligned_pairs | |----------|----------|-------------------| | afr | eng | 14549 | | afr | nbl | 6621 | | afr | nso | 15388 | | afr | sot | 8834 | | afr | ssw | 15610 | | afr | tsn | 12605 | | afr | tso | 14936 | | afr | ven | 5776 | | afr | xho | 16065 | | afr | zul | 14998 | | nbl | eng | 3616 | | nbl | nso | 6342 | | nbl | sot | 16163 | | nbl | ssw | 4655 | | nbl | tsn | 3369 | | nbl | tso | 4465 | | nbl | ven | 18984 | | nbl | xho | 5213 | | nbl | zul | 3868 | | nso | eng | 15257 | | nso | ssw | 18697 | | nso | tsn | 16179 | | nso | tso | 17617 | | nso | ven | 6367 | | sot | eng | 5212 | | sot | nso | 8077 | | sot | ssw | 5811 | | sot | tsn | 5450 | | sot | tso | 6586 | | sot | ven | 14098 | | ssw | eng | 15721 | | ssw | tso | 17880 | | ssw | ven | 4588 | | tsn | eng | 14544 | | tsn | ssw | 16386 | | tsn | tso | 16681 | | tsn | ven | 3267 | | tso | eng | 16068 | | ven | eng | 3670 | | ven | tso | 4578 | | xho | eng | 16537 | | xho | nso | 18110 | | xho | sot | 7489 | | xho | ssw | 18387 | | xho | tsn | 16571 | | xho | tso | 17954 | | xho | ven | 4559 | | xho | zul | 18145 | | zul | eng | 16149 | | zul | nso | 17630 | | zul | sot | 5975 | | zul | ssw | 18563 | | zul | tsn | 16482 | | zul | tso | 17789 | | zul | ven | 3606 | Authors ------- - Vukosi Marivate - [@vukosi](https://twitter.com/vukosi) - Matimba Shingange - Richard Lastrucci - Isheanesu Joseph Dzingirai - Jenalea Rajab Publications -------</pre> &gt; @inproceedings{lastrucci-etal-2023-preparing,<br> title = "Preparing the Vuk{'}uzenzele and {ZA}-gov-multilingual {S}outh {A}frican multilingual corpora",<br> author = "Richard Lastrucci and Isheanesu Dzingirai and Jenalea Rajab and Andani Madodonga and Matimba Shingange and Daniel Njini and Vukosi Marivate",<br> booktitle = "Proceedings of the Fourth workshop on Resources for African Indigenous Languages (RAIL 2023)",<br> month = may,<br> year = "2023",<br> address = "Dubrovnik, Croatia",<br> publisher = "Association for Computational Linguistics",<br> url = "https://aclanthology.org/2023.rail-1.3",<br> pages = "18--25"<br> }南非Gov-ZA多语言语料库 ============================== Github: https://github.com/dsfsi/gov-za-multilingual Zenodo: 关于本数据集 --------------------- 本数据集收录南非政府的内阁声明，数据从南非政府官方网站https://www.gov.za/cabinet-statements爬取获取。 本数据集包含11种语言的政府内阁声明，具体语言及对应代码如下表所示： | 语言名称 | 代码 | 语言名称 | 代码 | |------------|------|------------|------| | 英语（English） | (eng) | 北索托语（Sepedi） | (nso) | | 南非荷兰语（Afrikaans） | (afr) | 茨瓦纳语（Setswana） | (tsn) | | 恩德贝莱语（isiNdebele） | (nbl) | 斯威士兰语（Siswati） | (ssw) | | 科萨语（isiXhosa） | (xho) | 温达语（Tshivenda） | (ven) | | 祖鲁语（isiZulu） | (zul) | 聪加语（Xitstonga） | (tso) | | 塞索托语（Sesotho） | (sot) | | | 数据集的完整内容存储于JSON文件`/data/govza-cabinet-statements.json`中，同时提供按语言拆分的CSV文件，例如代表英语语料的`govza-cabinet-statements-en.csv`。 本数据集未包含Unicode或ASCII类特殊字符。请参阅`/data_statement.md`以获取完整的数据集相关信息。 *(待完成)* 余弦相似度得分≥0.65的对齐对数量 ------------------------------------------------------------ | 源语言代码 | 目标语言代码 | 对齐对数量 | |----------|----------|-------------------| | afr | eng | 14549 | | afr | nbl | 6621 | | afr | nso | 15388 | | afr | sot | 8834 | | afr | ssw | 15610 | | afr | tsn | 12605 | | afr | tso | 14936 | | afr | ven | 5776 | | afr | xho | 16065 | | afr | zul | 14998 | | nbl | eng | 3616 | | nbl | nso | 6342 | | nbl | sot | 16163 | | nbl | ssw | 4655 | | nbl | tsn | 3369 | | nbl | tso | 4465 | | nbl | ven | 18984 | | nbl | xho | 5213 | | nbl | zul | 3868 | | nso | eng | 15257 | | nso | ssw | 18697 | | nso | tsn | 16179 | | nso | tso | 17617 | | nso | ven | 6367 | | sot | eng | 5212 | | sot | nso | 8077 | | sot | ssw | 5811 | | sot | tsn | 5450 | | sot | tso | 6586 | | sot | ven | 14098 | | ssw | eng | 15721 | | ssw | tso | 17880 | | ssw | ven | 4588 | | tsn | eng | 14544 | | tsn | ssw | 16386 | | tsn | tso | 16681 | | tsn | ven | 3267 | | tso | eng | 16068 | | ven | eng | 3670 | | ven | tso | 4578 | | xho | eng | 16537 | | xho | nso | 18110 | | xho | sot | 7489 | | xho | ssw | 18387 | | xho | tsn | 16571 | | xho | tso | 17954 | | xho | ven | 4559 | | xho | zul | 18145 | | zul | eng | 16149 | | zul | nso | 17630 | | zul | sot | 5975 | | zul | ssw | 18563 | | zul | tsn | 16482 | | zul | tso | 17789 | | zul | ven | 3606 | 作者 ------- - Vukosi Marivate - [@vukosi](https://twitter.com/vukosi) - Matimba Shingange - Richard Lastrucci - Isheanesu Joseph Dzingirai - Jenalea Rajab 发表论文 ------- > @inproceedings{lastrucci-etal-2023-preparing,<br> > title = "《筹备Vuk'uzenzele与ZA-gov-multilingual南非多语言语料库》",<br> > author = "Richard Lastrucci、Isheanesu Dzingirai、Jenalea Rajab、Andani Madodonga、Matimba Shingange、Daniel Njini、Vukosi Marivate",<br> > booktitle = "第四届非洲本土语言资源研讨会（RAIL 2023）论文集",<br> > month = 五月,<br> > year = "2023",<br> > address = "克罗地亚杜布罗夫尼克",<br> > publisher = "国际计算语言学协会（Association for Computational Linguistics）",<br> > url = "https://aclanthology.org/2023.rail-1.3",<br> > pages = "18–25" > }