Trelis/libritts-bpe-tokens

libritts-bpe-tokens数据集是LibriTTS-R数据集中文本归一化转录的GPT-2 BPE（字节对编码）令牌集合。每个话语以EOS令牌（50256）结束，令牌以列表形式存储在`token_ids`列中，词汇表大小为50,257。数据集包含约538小时的音频内容，分拆为train.clean.100（约32,000个话语、53小时）、train.clean.360（约112,000个话语、218小时）、train.other.500（约250,000个话语、258小时）和dev.clean（约5,600个话语、9小时）四个部分，确保训练集和开发集之间没有说话者重叠。数据列包括id（源话语标识符）、speaker（说话者id）、text（归一化文本）和token_ids（令牌列表）。数据集基于CC-BY-4.0许可证，与LibriTTS-R源数据一致，GPT-2令牌化器使用MIT许可证。

GPT-2 BPE tokens of LibriTTS-R `text_normalized` transcripts. Each utterance is terminated with the EOS token (50256). Tokens are in column `token_ids` (list[int]), vocab=50,257. The dataset includes approximately 538 hours of audio content, split into train.clean.100 (~32k utterances, ~53 hours), train.clean.360 (~112k utterances, ~218 hours), train.other.500 (~250k utterances, ~258 hours), and dev.clean (~5.6k utterances, ~9 hours) splits, with structural disjointness ensuring no speaker overlap between train and dev sets. Columns include id (source utterance identifier), speaker (speaker id), text (normalized text), and token_ids (list of tokens). Licensed under CC-BY-4.0, matching the LibriTTS-R source, with GPT-2 tokenizer under MIT license.