klokedm/MaCoCu-sl-tokenized

该数据集提供了斯洛文尼亚网络语料库MaCoCu的预标记版本，包括来自各种大型语言模型标记器的标记ID和标记计数。此数据集旨在方便研究和实验，提供了准备好的标记数据，节省了在重复设置过程中计算资源。数据集包含文本数据和元数据，并且针对不同模型提供了具体的标记化细节。此外，还提供了额外的统计信息，如句数、段落数、字符总数和词数。数据集仅包含训练集。

This dataset provides a pre-tokenized version of the Slovene web corpus MaCoCu, including token IDs and token counts generated by various large language model tokenizers. The dataset is intended for research and experimentation, providing ready-to-use tokenized data to save computational resources during repeated setups. It contains text data and metadata, with specific tokenization details for different models. Additional statistics such as sentence count, paragraph count, character length, and word count are also provided. The dataset includes only the training split.