LDFQ/wikitext

WikiText语言建模数据集是一个从维基百科上经过验证的优秀和特色文章中提取的超过1亿个标记的集合。该数据集在知识共享署名-相同方式共享许可下可用。与经过预处理的Penn Treebank (PTB)版本相比，WikiText-2的规模是其2倍以上，而WikiText-103的规模是其110倍以上。WikiText数据集还具有更大的词汇量，并保留了原始的大小写、标点符号和数字——这些在PTB中都被移除。由于它由完整的文章组成，该数据集非常适合能够利用长期依赖关系的模型。每个子集都有两种不同的变体：- 原始版本（用于字符级工作）包含添加<unk>（未知）标记之前的原始标记。- 非原始版本（用于词级工作）仅包含其词汇表中的标记（wiki.train.tokens、wiki.valid.tokens和wiki.test.tokens）。词汇表外的标记已被替换为<unk>标记。The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License. Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies. Each subset comes in two different variants: - Raw (for character level work) contain the raw tokens, before the addition of the <unk> (unknown) tokens. - Non-raw (for word level work) contain only the tokens in their vocabulary (wiki.train.tokens, wiki.valid.tokens, and wiki.test.tokens). The out-of-vocabulary tokens have been replaced with the the <unk> token.