betteib/tokenized_1m

该数据集包含了文本处理任务中常用的三个特征字段：input_ids、attention_mask和labels。input_ids是int32类型的序列，代表了输入文本的token ID；attention_mask是int8类型的序列，用于指示输入序列中哪些token是有效的；labels是int64类型的序列，代表了文本的标签或分类。数据集分为训练集和验证集，其中训练集包含1,353,321个示例，大小为9.02GB；验证集包含39,510个示例，大小为0.26GB。整个数据集的总大小为9.29GB，下载大小为154MB。

The dataset includes three common feature fields for text processing tasks: input_ids, attention_mask, and labels. input_ids are sequences of int32 type representing the token IDs of the input text; attention_mask are sequences of int8 type indicating which tokens in the input sequence are valid; labels are sequences of int64 type representing the text labels or categories. The dataset is split into a training set and a validation set, with the training set containing 1,353,321 examples and being 9.02GB in size, and the validation set containing 39,510 examples and being 0.26GB in size. The total size of the entire dataset is 9.29GB, with a download size of 154MB.