answer722qzd/tokenized_datasets

---
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: valid
path: data/valid-*
dataset_info:
features:
- name: input_ids
sequence: int32
splits:
- name: train
num_bytes: 8618263476
num_examples: 16702061
- name: valid
num_bytes: 48072624
num_examples: 93164
download_size: 3804670335
dataset_size: 8666336100
---
# Dataset Card for "tokenized_datasets"

[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

The dataset named tokenized_datasets contains tokenized data used for training and validating models. It is divided into a training set with 16,702,061 examples and a validation set with 93,164 examples. The main feature of the dataset is input_ids, which represents the integer sequence of tokens.

原始信息汇总

数据集概述

配置信息

• 默认配置 (config_name: default):
  • 训练数据 (split: train): 路径为 data/train-*
  • 验证数据 (split: valid): 路径为 data/valid-*

数据集特征

• 特征名称: input_ids
• 数据类型: int32

数据集分割

• 训练集 (split: train):
  • 样本数量: 16702061
  • 数据大小: 8618263476 字节
• 验证集 (split: valid):
  • 样本数量: 93164
  • 数据大小: 48072624 字节

数据集大小

• 下载大小: 3804670335 字节
• 总数据集大小: 8666336100 字节