libritts-bpe-tokens

# libritts-bpe-tokens > To learn about Trelis Enterprise Voice Services, see [Trelis.com/voice-ai-services](https://Trelis.com/voice-ai-services). GPT-2 BPE tokens of LibriTTS-R `text_normalized` transcripts. Each utterance is terminated with the EOS token (50256). Tokens are in column `token_ids` (list[int]), vocab=50,257. ## Splits Mirrors the source LibriTTS-R splits (filtered by parler-tts; total ≈ 538 h): | split | utterances | hours | |---|---:|---:| | `train.clean.100` | ~32 k | ~53 h | | `train.clean.360` | ~112 k | ~218 h | | `train.other.500` | ~250 k | ~258 h | | `dev.clean` | ~5.6 k | ~9 h | Source: [`parler-tts/libritts_r_filtered`](https://huggingface.co/datasets/parler-tts/libritts_r_filtered). Disjointness between splits is structural (HF split definition — no speaker overlap between train.* and dev.*). ## Schema (one row per utterance) | column | type | description | |---|---|---| | `id` | string | Source utterance id (LibriTTS `speaker_chapter_segment`) | | `speaker` | string | LibriTTS speaker id | | `text` | string | `text_normalized` from source | | `token_ids` | list[uint32] | GPT-2 BPE tokens + EOS, vocab 50,257 | ## Loading ```python from datasets import load_dataset # Load the whole dataset (all 4 splits) ds = load_dataset("Trelis/libritts-bpe-tokens") # Just one split clean_360 = load_dataset("Trelis/libritts-bpe-tokens", split="train.clean.360") # Combine all train splits all_train = load_dataset("Trelis/libritts-bpe-tokens", split="train.clean.100+train.clean.360+train.other.500") print(all_train[0]) ``` ## Companion datasets (same audio content, different tokenization) - `Trelis/libritts-mimi-tokens` - `Trelis/libritts-snac-tokens` - `Trelis/libritts-neucodec-tokens` ## Reproducing Encoder code: [TrelisResearch/audio-bits](https://github.com/TrelisResearch/audio-bits) (see `scripts/encode_codec.py` and `scripts/tokenize_text.py`). **Audio prep**: source LibriTTS-R is 24 kHz. NeuCodec (16 kHz) is downsampled with `torchaudio.functional.resample(method="sinc_interp_kaiser")` (polyphase, anti-aliased). Mimi and SNAC consume 24 kHz directly. Utterances > 20 s are truncated to 20 s of audio (text is kept full). **Per-row token count** = `min(ceil(n_audio_samples * T / max_padded_samples), T)` where `T` is the codec's per-batch output length, so the count tracks the model's own time grid and is exact for fixed-stride encoders. ## License CC-BY-4.0 (matches LibriTTS-R source). GPT-2 tokenizer (MIT).