athena-ml/gpt4_short_stories_with_tokens

该数据集包含两个主要特征：text和tokens，分别表示原始文本内容和分词后的序列。数据集分为训练集和测试集，训练集包含2,119,719个样本，占用5,225,576,523字节；测试集包含21,990个样本，占用52,717,436字节。数据集的下载大小为1,802,972,998字节，总大小为5,278,293,959字节。配置部分指定了训练集和测试集的文件路径。

The dataset contains two main features: text and tokens, representing the text content and tokenized form of the text, respectively. The dataset is divided into a training set and a test set, containing 2119719 and 21990 samples, respectively. The total download size of the dataset is 1802972998 bytes, and the total dataset size is 5278293959 bytes.