Ba2han/mixed-tokenized_2811

---
dataset_info:
features:
- name: input_ids
list: int32
splits:
- name: train_chunk_1
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_2
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_3
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_4
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_5
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_6
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_7
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_8
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_9
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_10
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_11
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_12
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_13
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_14
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_15
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_16
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_17
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_18
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_19
num_bytes: 3274400000
num_examples: 200000
- name: train_chunk_20
num_bytes: 1884237108
num_examples: 115089
download_size: 26359338785
dataset_size: 64097837108
configs:
- config_name: default
data_files:
- split: train_chunk_1
path: data/train_chunk_1-*
- split: train_chunk_2
path: data/train_chunk_2-*
- split: train_chunk_3
path: data/train_chunk_3-*
- split: train_chunk_4
path: data/train_chunk_4-*
- split: train_chunk_5
path: data/train_chunk_5-*
- split: train_chunk_6
path: data/train_chunk_6-*
- split: train_chunk_7
path: data/train_chunk_7-*
- split: train_chunk_8
path: data/train_chunk_8-*
- split: train_chunk_9
path: data/train_chunk_9-*
- split: train_chunk_10
path: data/train_chunk_10-*
- split: train_chunk_11
path: data/train_chunk_11-*
- split: train_chunk_12
path: data/train_chunk_12-*
- split: train_chunk_13
path: data/train_chunk_13-*
- split: train_chunk_14
path: data/train_chunk_14-*
- split: train_chunk_15
path: data/train_chunk_15-*
- split: train_chunk_16
path: data/train_chunk_16-*
- split: train_chunk_17
path: data/train_chunk_17-*
- split: train_chunk_18
path: data/train_chunk_18-*
- split: train_chunk_19
path: data/train_chunk_19-*
- split: train_chunk_20
path: data/train_chunk_20-*
---

数据集信息：
特征：
- 名称：输入令牌ID（input_ids），类型为32位整数（int32）组成的列表
数据划分：
- 划分名称：train_chunk_1，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_2，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_3，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_4，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_5，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_6，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_7，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_8，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_9，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_10，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_11，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_12，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_13，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_14，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_15，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_16，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_17，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_18，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_19，字节占用量：3274400000，样本数量：200000
- 划分名称：train_chunk_20，字节占用量：1884237108，样本数量：115089
下载总大小：26359338785
数据集总大小：64097837108
配置项：
- 配置名称：default，数据文件映射：
- 划分：train_chunk_1，文件路径：data/train_chunk_1-*
- 划分：train_chunk_2，文件路径：data/train_chunk_2-*
- 划分：train_chunk_3，文件路径：data/train_chunk_3-*
- 划分：train_chunk_4，文件路径：data/train_chunk_4-*
- 划分：train_chunk_5，文件路径：data/train_chunk_5-*
- 划分：train_chunk_6，文件路径：data/train_chunk_6-*
- 划分：train_chunk_7，文件路径：data/train_chunk_7-*
- 划分：train_chunk_8，文件路径：data/train_chunk_8-*
- 划分：train_chunk_9，文件路径：data/train_chunk_9-*
- 划分：train_chunk_10，文件路径：data/train_chunk_10-*
- 划分：train_chunk_11，文件路径：data/train_chunk_11-*
- 划分：train_chunk_12，文件路径：data/train_chunk_12-*
- 划分：train_chunk_13，文件路径：data/train_chunk_13-*
- 划分：train_chunk_14，文件路径：data/train_chunk_14-*
- 划分：train_chunk_15，文件路径：data/train_chunk_15-*
- 划分：train_chunk_16，文件路径：data/train_chunk_16-*
- 划分：train_chunk_17，文件路径：data/train_chunk_17-*
- 划分：train_chunk_18，文件路径：data/train_chunk_18-*
- 划分：train_chunk_19，文件路径：data/train_chunk_19-*
- 划分：train_chunk_20，文件路径：data/train_chunk_20-*