TemryL/tokenized_wikipedia_20220301.en_train_128

该数据集包含从2022年3月1日的英文维基百科转储中提取的文本块，每个文本块经过BERT base uncased分词器处理，并被分割成128个token的块。每个块包含token IDs、attention mask、文档索引和块索引等信息。数据集的创建过程包括加载原始维基百科数据集、使用BERT分词器进行分词、将分词后的文章处理成128个token的块，并保留每个块的文档索引和块索引。

This dataset contains tokenized chunks of text from the English Wikipedia dump of March 1, 2022. Each entry in the dataset represents a chunk of text from Wikipedia, with information about which document and position within the document it comes from. The dataset creation process involves loading the raw Wikipedia dataset, tokenizing articles using the BERT base uncased tokenizer, processing the tokenized articles to create chunks of exactly 128 tokens each, and preserving document index and chunk index information. Each row in the dataset corresponds to a chunk of text from a Wikipedia document and contains fields such as token_ids, attention_mask, doc_idx, and chunk_idx.