anrilombard/mzansi-text-tokenized

--- language: - af - en - nso - sot - ssw - tsn - tso - ven - xho - zul - nbl tags: - pretraining - tokenized - south-african-languages - multilingual - mzansitext license: apache-2.0 --- # MzansiText Tokenized Ready-to-train tokenized version of **MzansiText**, chunked to a context length of 2048 tokens. [](https://github.com/Anri-Lombard/sallm) [](https://arxiv.org/abs/2603.20732) [](https://huggingface.co/anrilombard/mzansilm-125m) [](https://huggingface.co/collections/anrilombard/mzansilm-69635ca7b60efedb9dfcb09e) ## Dataset Details - Tokenizer: custom BPE, `65536` vocabulary - Chunking: `2048` tokens per example with EOS separators between documents - Schema: ```json { "input_ids": ["int"], "lang": "string" } ``` ### Split Sizes | Split | Examples | |---|---:| | Train | 3,943,584 | | Validation | 19,379 | | Test | 19,341 | ## Usage ```python from datasets import load_dataset ds = load_dataset("anrilombard/mzansi-text-tokenized", split="train") print(ds[0].keys()) ``` ## Related Releases - Paper: [arXiv:2603.20732](https://arxiv.org/abs/2603.20732) - Model: [anrilombard/mzansilm-125m](https://huggingface.co/anrilombard/mzansilm-125m) - Raw corpus: [anrilombard/mzansi-text](https://huggingface.co/datasets/anrilombard/mzansi-text) - GitHub code and configs: [https://github.com/Anri-Lombard/sallm](https://github.com/Anri-Lombard/sallm) Full preprocessing pipeline (including this exact cleaning script) is in [`data/cleaning/`](https://github.com/Anri-Lombard/sallm/tree/main/data/cleaning) on GitHub. ## Citation Please cite the paper: ```bibtex @misc{lombard2026mzansitextmzansilmopencorpus, title={MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages}, author={Anri Lombard and Simbarashe Mawere and Temi Aina and Ethan Wolff and Sbonelo Gumede and Elan Novick and Francois Meyer and Jan Buys}, year={2026}, eprint={2603.20732}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2603.20732}, } ``` ## License Apache License 2.0--- language: - 南非荷兰语（Afrikaans） - 英语（English） - 北索托语（Northern Sotho） - 南索托语（Southern Sotho） - 斯威士语（Swati） - 茨瓦纳语（Tswana） - 聪加语（Tsonga） - 文达语（Venda） - 科萨语（Xhosa） - 祖鲁语（Zulu） - 南恩德贝莱语（Southern Ndebele） tags: - 预训练 - 已分词 - 南非语言 - 多语言 - mzansitext license: apache-2.0 --- # MzansiText 分词版 **MzansiText**的可直接用于训练的分词版本，已被切分为上下文长度为2048个Token的样本块。 [](https://github.com/Anri-Lombard/sallm) [](https://arxiv.org/abs/2603.20732) [](https://huggingface.co/anrilombard/mzansilm-125m) [](https://huggingface.co/collections/anrilombard/mzansilm-69635ca7b60efedb9dfcb09e) ## 数据集详情 - 分词器：自定义字节对编码（BPE），词汇表规模为65536 - 分块策略：每个样本包含2048个Token，文档间以结束符（EOS）作为分隔符 - 数据结构： json { "input_ids": ["int"], "lang": "string" } ### 数据集划分规模 | 数据集划分 | 样本数量 | |---|---:| | 训练集 | 3,943,584 | | 验证集 | 19,379 | | 测试集 | 19,341 | ## 使用方法 python from datasets import load_dataset ds = load_dataset("anrilombard/mzansi-text-tokenized", split="train") print(ds[0].keys()) ## 相关发布项目 - 论文：[arXiv:2603.20732](https://arxiv.org/abs/2603.20732) - 模型：[anrilombard/mzansilm-125m](https://huggingface.co/anrilombard/mzansilm-125m) - 原始语料库：[anrilombard/mzansi-text](https://huggingface.co/datasets/anrilombard/mzansi-text) - GitHub代码与配置文件：[https://github.com/Anri-Lombard/sallm](https://github.com/Anri-Lombard/sallm) 完整的预处理流水线（包含本数据集所用的清洗脚本）可在GitHub仓库的[`data/cleaning/`](https://github.com/Anri-Lombard/sallm/tree/main/data/cleaning)目录中获取。 ## 引用方式 请引用如下论文： bibtex @misc{lombard2026mzansitextmzansilmopencorpus, title={MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages}, author={Anri Lombard and Simbarashe Mawere and Temi Aina and Ethan Wolff and Sbonelo Gumede and Elan Novick and Francois Meyer and Jan Buys}, year={2026}, eprint={2603.20732}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2603.20732}, } ## 许可证 Apache许可证2.0