The South African Gov-ZA multilingual corpus

The South African Gov-ZA multilingual corpus
==============================
Github: https://github.com/dsfsi/gov-za-multilingual
Zenodo:

About Dataset
---------------------
The data set contains cabinet statements from the South African government. Data was scraped from the governments website:
https://www.gov.za/cabinet-statements

The datasets contain government cabinet statements in 11 languages, namely:

| Language | Code | Language | Code |
|------------|------|------------|------|
| English | (eng) | Sepedi | (nso) |
| Afrikaans | (afr) | Setswana | (tsn) |
| isiNdebele | (nbl) | Siswati | (ssw) |
| isiXhosa | (xho) | Tshivenda | (ven) |
| isiZulu | (zul) | Xitstonga | (tso) |
| Sesotho | (sot) |

The dataset contains the full data in a JSON file (/data/govza-cabinet-statements.json), as well as CSV’s split by each language, eg: “govza-cabinet-statements-en.csv” for english.
The dataset does not contain special characters like unicode or ascii.

Please see the [data-statement.md](/data_statement.md) for full dataset information. *(TODO)*

Number of Aligned Pairs with Cosine Similarity Score >= 0.65
------------------------------------------------------------

| src_lang | trg_lang | num_aligned_pairs |
|----------|----------|-------------------|
| afr | eng | 14549 |
| afr | nbl | 6621 |
| afr | nso | 15388 |
| afr | sot | 8834 |
| afr | ssw | 15610 |
| afr | tsn | 12605 |
| afr | tso | 14936 |
| afr | ven | 5776 |
| afr | xho | 16065 |
| afr | zul | 14998 |
| nbl | eng | 3616 |
| nbl | nso | 6342 |
| nbl | sot | 16163 |
| nbl | ssw | 4655 |
| nbl | tsn | 3369 |
| nbl | tso | 4465 |
| nbl | ven | 18984 |
| nbl | xho | 5213 |
| nbl | zul | 3868 |
| nso | eng | 15257 |
| nso | ssw | 18697 |
| nso | tsn | 16179 |
| nso | tso | 17617 |
| nso | ven | 6367 |
| sot | eng | 5212 |
| sot | nso | 8077 |
| sot | ssw | 5811 |
| sot | tsn | 5450 |
| sot | tso | 6586 |
| sot | ven | 14098 |
| ssw | eng | 15721 |
| ssw | tso | 17880 |
| ssw | ven | 4588 |
| tsn | eng | 14544 |
| tsn | ssw | 16386 |
| tsn | tso | 16681 |
| tsn | ven | 3267 |
| tso | eng | 16068 |
| ven | eng | 3670 |
| ven | tso | 4578 |
| xho | eng | 16537 |
| xho | nso | 18110 |
| xho | sot | 7489 |
| xho | ssw | 18387 |
| xho | tsn | 16571 |
| xho | tso | 17954 |
| xho | ven | 4559 |
| xho | zul | 18145 |
| zul | eng | 16149 |
| zul | nso | 17630 |
| zul | sot | 5975 |
| zul | ssw | 18563 |
| zul | tsn | 16482 |
| zul | tso | 17789 |
| zul | ven | 3606 |

Authors
-------
- Vukosi Marivate - [@vukosi](https://twitter.com/vukosi)
- Matimba Shingange
- Richard Lastrucci
- Isheanesu Joseph Dzingirai
- Jenalea Rajab

Publications
-------

> @inproceedings{lastrucci-etal-2023-preparing,
    title = "Preparing the Vuk{'}uzenzele and {ZA}-gov-multilingual {S}outh {A}frican multilingual corpora",
    author = "Richard Lastrucci and Isheanesu Dzingirai and Jenalea Rajab and Andani Madodonga and Matimba Shingange and Daniel Njini and Vukosi Marivate",
    booktitle = "Proceedings of the Fourth workshop on Resources for African Indigenous Languages (RAIL 2023)",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.rail-1.3",
    pages = "18--25"
}