Aarushhh/wikipedia-simple-splitted

该数据集名为Wikipedia simple splitted，使用了Langchain的RecursiveCharacterTextSplitter进行分割。数据集有两个配置，分别是max_len-1024和max_len-448，每个配置包含一个训练集，数据特征为文本字符串。数据集的特点是其分割方式不同于简单的按长度分割，而是使用RecursiveCharacterTextSplitter，确保分割的块不会随机结束。此外，数据集可以用于Annealing LLMs，并且支持非常大的批量大小。

This dataset is sourced from the simple version of Wikipedia and is split using Langchains RecursiveCharacterTextSplitter. The dataset is unique in that it does not split by fixed length, but rather uses RecursiveCharacterTextSplitter to ensure that the chunks do not end randomly. The dataset has two configurations, supporting text chunks with maximum lengths of 1024 and 448 characters. It is suitable for training large language models (LLMs), particularly for Annealing LLMs.