bethea/cc-tokenized

该数据集包含三个主要特性：input_ids、attention_mask和labels，分别表示输入ID序列、注意力掩码序列和标签序列。数据集分为训练集、验证集和测试集，训练集包含800,000个示例，验证集和测试集各包含100,000个示例。数据集的下载大小为653,864,254字节，总大小为6,668,000,000字节。配置信息指定了每个分割对应的数据文件路径。

This dataset is primarily used for natural language processing tasks, featuring three core components: input_ids for storing encoded input sequences, attention_mask for indicating valid parts of the sequence, and labels for storing target labels. The dataset is divided into training, validation, and test sets, containing 800,000, 100,000, and 100,000 samples respectively. The total size and download size of the dataset are 6,668,000,000 bytes and 653,864,254 bytes, respectively.

原始信息汇总

数据集概述

特征

• input_ids: 序列类型为 int32
• attention_mask: 序列类型为 int8
• labels: 序列类型为 int64

数据分割

• train:
  • 字节数: 5334400000
  • 样本数: 800000
• validation:
  • 字节数: 666800000
  • 样本数: 100000
• test:
  • 字节数: 666800000
  • 样本数: 100000

数据集大小

• 下载大小: 653864254 字节
• 数据集总大小: 6668000000 字节

配置

• config_name: default
  • data_files:
    • train: data/train-*
    • validation: data/validation-*
    • test: data/test-*