SemantikaEU/MaCoCu-sl-tokenized

该数据集提供了斯洛文尼亚网络语料库MaCoCu的预分词版本，包括原始文本数据和元数据，并增加了由多种流行的大型语言模型分词器生成的token IDs和token计数。旨在通过提供即用型分词数据，方便机器学习框架的使用，节省计算资源。

This dataset provides a pre-tokenized version of the Slovene web corpus MaCoCu, including the original text data and metadata, augmented with token IDs and token counts generated by several popular large language model tokenizers. The goal is to facilitate research and experimentation by providing ready-to-use tokenized data, saving computational resources during repeated setups.