Avelina/smollm-corpus

SmolLM-Corpus是一个重新排列和分片的数据集，包含cosmopedia-v2和fineweb-edu-dedup两个子集，以及从python-edu仓库提取的python-edu子集。数据集被分为24个子目录，包含23698个分片的压缩jsonl文件，每个文件包含文本数据。适用于文本生成任务，数据集语言为英语，大小介于100M到1B之间。

SmolLM-Corpus is a reorganized and sharded dataset consisting of the cosmopedia-v2 and fineweb-edu-dedup subsets, as well as the python-edu subset from the python-edu repository. The dataset is divided into 24 subdirectories, containing 23,698 sharded compressed jsonl files, each containing textual data. It is suitable for text generation tasks, with the dataset language being English and the size ranging between 100M and 1B.