nineninesix/emolia_filtered_nano_codec_21_dataset

--- configs: - config_name: default data_files: - split: train path: data/train-* dataset_info: features: - name: id dtype: string - name: text dtype: string - name: speaker dtype: string - name: language dtype: string - name: nano_layer_1 list: int64 - name: nano_layer_2 list: int64 - name: nano_layer_3 list: int64 - name: nano_layer_4 list: int64 - name: nano_layer_5 list: int64 - name: nano_layer_6 list: int64 - name: nano_layer_7 list: int64 - name: nano_layer_8 list: int64 - name: encoded_len dtype: int64 splits: - name: train num_bytes: 68922549168 num_examples: 5397323 download_size: 68591078406 dataset_size: 68922549168 license: apache-2.0 pretty_name: Emolia · Filtered · NanoCodec (FSQ) Tokens task_categories: - text-to-speech - audio-to-audio size_categories: - 1M<n<10M tags: - text-to-speech - tts - speech - audio-codec - neural-codec - nanocodec - fsq - discrete-tokens - emolia - emilia - speech-synthesis --- # Emolia · Filtered · NanoCodec (FSQ) Tokens A cleaned, pre-tokenized version of [**laion/Emolia**](https://huggingface.co/datasets/laion/Emolia) prepared for **text-to-speech (TTS)** training. The pipeline is two steps: 1. **Quality filtering** with the open-source [`audio_filter`](https://github.com/KaniTTS-research-team/audio_filter) tool — this removes the dirtiest recordings (noise, clipping, band-limiting, robotic artifacts, overlapping speakers), which matters a lot for TTS quality. 2. **Discrete audio tokenization** with NVIDIA [`nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps`](https://huggingface.co/nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps) (an **FSQ** neural audio codec) — every waveform is replaced by 8 layers of integer codes. The result is a compact, ready-to-train dataset: **text + speaker + language + audio codec tokens**. No raw audio is redistributed here — only discrete codec indices. - **Examples:** 5,397,323 (single `train` split) - **On-disk size:** ~68.9 GB (Apache Arrow, 138 shards) - **Audio codec:** NanoCodec, 22 kHz, 1.89 kbps, 21.5 frames/sec, 8 FSQ codebooks - **Source dataset:** [laion/Emolia](https://huggingface.co/datasets/laion/Emolia) (CC-BY-4.0) --- ## Dataset structure Each row is one utterance. There is no audio blob — the audio is encoded as 8 parallel streams of NanoCodec indices (`nano_layer_1` … `nano_layer_8`), each of length `encoded_len`. | Column | Type | Description | | --------------- | ------------- | ----------------------------------------------------------------- | | `id` | `string` | Unique utterance id (carried over from Emolia) | | `text` | `string` | Transcript of the utterance | | `speaker` | `string` | Speaker id | | `language` | `string` | Language code (`en`, `zh`, `de`, `fr`, `ja`, `ko`) | | `nano_layer_1` | `list<int64>` | NanoCodec FSQ codes, codebook 1 — length `encoded_len` | | `nano_layer_2` | `list<int64>` | NanoCodec FSQ codes, codebook 2 | | `nano_layer_3` | `list<int64>` | NanoCodec FSQ codes, codebook 3 | | `nano_layer_4` | `list<int64>` | NanoCodec FSQ codes, codebook 4 | | `nano_layer_5` | `list<int64>` | NanoCodec FSQ codes, codebook 5 | | `nano_layer_6` | `list<int64>` | NanoCodec FSQ codes, codebook 6 | | `nano_layer_7` | `list<int64>` | NanoCodec FSQ codes, codebook 7 | | `nano_layer_8` | `list<int64>` | NanoCodec FSQ codes, codebook 8 | | `encoded_len` | `int64` | Number of codec frames (≈ duration_seconds × 21.5) | > The 8 `nano_layer_*` columns are aligned frame-by-frame: index `t` of every layer > describes the same 1/21.5 s ≈ 46 ms audio frame. --- ## Usage ### Load the dataset ```python from datasets import load_dataset # Streaming is recommended — the dataset is large. ds = load_dataset("nineninesix/emolia_filtered_nano_codec_21_dataset", split="train", streaming=True) sample = next(iter(ds)) print(sample["text"], sample["language"], sample["speaker"]) print("frames:", sample["encoded_len"]) print("codebook 1:", sample["nano_layer_1"][:10]) ``` ### Reconstruct audio from the tokens The codes can be decoded back to a 22 kHz waveform with the matching NeMo NanoCodec model. ```python import torch from nemo.collections.tts.models import AudioCodecModel codec = AudioCodecModel.from_pretrained("nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps") codec.eval() # Stack the 8 layers into (Batch=1, Codebooks=8, Time) layers = [sample[f"nano_layer_{i}"] for i in range(1, 9)] tokens = torch.tensor(layers, dtype=torch.long).unsqueeze(0) # (1, 8, T) tokens_len = torch.tensor([sample["encoded_len"]], dtype=torch.long) # (1,) with torch.no_grad(): audio, audio_len = codec.decode(tokens=tokens, tokens_len=tokens_len) # audio -> 22 kHz waveform, save with soundfile/torchaudio ``` --- ## How this dataset was built ### 1. Quality filtering — [`audio_filter`](https://github.com/KaniTTS-research-team/audio_filter) We ran every Emolia recording through the open-source [`audio_filter`](https://github.com/KaniTTS-research-team/audio_filter) pipeline and kept only the clean ("good") audio. The tool analyzes acoustic / spectral characteristics and assigns each file a verdict of `good`, `uncertain`, or `bad` in two sequential stages: - **Acoustic quality screening** — rejects noisy, clipped, robotic, or bandwidth-limited recordings using spectral degradation metrics. Two models are used depending on the audio bandwidth: - **V1** for narrowband audio (≤ 24 kHz), scoring 12 acoustic metrics; - **V2** for wideband audio (> 24 kHz), scoring 34 acoustic metrics with loudness normalization. - **Speaker-overlap detection** — a Pyannote segmentation model flags files that contain simultaneous speech from multiple speakers; this runs only on files that already passed the quality stage. Only recordings that passed both stages ("good") were retained for tokenization. This step is what removes the "dirtiest" audio and is critical for downstream TTS quality. > Reproduce the filtering with the exact tool and thresholds documented in the repo: > <https://github.com/KaniTTS-research-team/audio_filter> ### 2. Tokenization — NVIDIA NanoCodec (FSQ) The surviving clean audio was encoded with [`nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps`](https://huggingface.co/nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps), a **Finite Scalar Quantization (FSQ)** neural audio codec: - **Sample rate:** 22 kHz - **Bitrate:** 1.89 kbps - **Frame rate:** 21.5 frames/second - **Codebooks:** 8 (stored as `nano_layer_1` … `nano_layer_8`) Each waveform becomes 8 aligned integer streams plus `encoded_len` (the number of frames). Raw audio is **not** included — only these discrete indices, which keeps the dataset compact and avoids redistributing the source audio. --- ## Provenance & derivation This dataset is a **derivative** of [**laion/Emolia**](https://huggingface.co/datasets/laion/Emolia), an emotion-enriched multilingual speech dataset built on top of **Emilia** (and its YODAS subset). We did **not** change the transcripts, speaker ids, or language labels — we only: 1. **Filtered out** low-quality / overlapping-speaker recordings, and 2. **Replaced the audio waveforms** with NanoCodec (FSQ) discrete tokens. For details about the original annotations, emotion labels, and speaker embeddings, see the [Emolia dataset card](https://huggingface.co/datasets/laion/Emolia). --- ## Licensing & attribution - **This packaging (README, filtering/tokenization pipeline, and the code representation of the data) is released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).** - The **underlying speech content is derived from [laion/Emolia](https://huggingface.co/datasets/laion/Emolia), which is distributed under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/).** Attribution to LAION / the Emolia and Emilia authors is required, and you must comply with CC-BY-4.0 (and the original Emilia terms) when using the audio-derived content in this dataset. By using this dataset you agree to respect both the Apache-2.0 terms of this repository and the CC-BY-4.0 terms of the upstream Emolia data. --- ## Citation If you use this dataset, please cite the upstream Emolia / Emilia work and this repository. ```bibtex @misc{emolia_filtered_nanocodec, title = {Emolia · Filtered · NanoCodec (FSQ) Tokens}, author = {Nineninesix, Inc.}, year = {2026}, howpublished = {\url{https://huggingface.co/datasets/nineninesix/emolia_filtered_nano_codec_21_dataset}}, note = {Filtered and NanoCodec-tokenized derivative of laion/Emolia} } ``` Please also cite the source dataset [laion/Emolia](https://huggingface.co/datasets/laion/Emolia) and the NVIDIA NanoCodec model. --- ## Acknowledgements - [LAION](https://laion.ai/) and the Emolia / Emilia authors for the source dataset. - [NVIDIA NeMo](https://github.com/NVIDIA/NeMo) for the NanoCodec (FSQ) audio codec. - [`audio_filter`](https://github.com/KaniTTS-research-team/audio_filter) for the quality-filtering pipeline.A cleaned, pre-tokenized version of the Emolia dataset prepared for text-to-speech (TTS) training. The dataset is processed through two steps: first, quality filtering using the open-source audio_filter tool to remove low-quality recordings such as noise, clipping, band-limiting, robotic artifacts, and overlapping speakers; second, discrete audio tokenization with NVIDIA NanoCodec (an FSQ neural audio codec) that replaces each waveform with 8 layers of integer codes. The result is a compact, ready-to-train dataset containing text, speaker, language, and audio codec tokens, with no raw audio included. It includes 5,397,323 examples, has an on-disk size of approximately 68.9 GB, uses the NanoCodec audio codec at 22 kHz sample rate, 1.89 kbps bitrate, 21.5 frames per second, and supports multiple languages such as English, Chinese, German, French, Japanese, and Korean.

搜集汇总

数据集介绍

构建方式

该数据集的构建分为两个严谨的阶段。首先，源自laion/Emolia的原始语音数据经由开源音频过滤器audio_filter进行质量筛查，该工具通过分析声学与频谱特征，依次执行声学质量筛选与说话人重叠检测，仅保留两阶段均判定为“良好”的干净录音。随后，幸存的高质量音频被送入NVIDIA NanoCodec（FSQ）神经音频编解码器，在22 kHz采样率下以每声道8个码本、21.5帧/秒的速率，将每个波形完全替换为8层对齐的整数编码序列，最终形成不含原始音频的紧凑型语料库。

特点

本数据集的核心特色在于其极致的紧凑性与即训即用能力。全部约540万条语音样本均以离散令牌形式存储，8层FSQ码本跨越编码长度对齐，彻底消除了原始音频的冗余，节省了约68.9 GB的存储空间。此外，数据保留了原始语料的文本转录、说话人标识与六种语言标签（含中、英、德、法、日、韩），并额外指明了每段音频的编码帧数，为文本转语音（TTS）模型的训练提供了完整且可直接投入使用的结构化输入。

使用方法

使用者可通过Hugging Face Datasets库以惰性流式加载方式访问该数据集，建议采用streaming=True参数以应对大容量场景。加载后每条样本包含文本、说话人、语言及8层纳秒码流，这些离散令牌可配合NVIDIA NeMo中的NanoCodec解码器还原为22 kHz波形，仅需将各层堆叠为(1,8,T)张量并调用codec.decode函数即可。值得注意的是，数据不包含原始音频，所有操作均围绕整型编码展开，极大简化了训练流水线的构建。

背景与挑战

背景概述

在神经音频编解码技术蓬勃发展的背景下，文本到语音（TTS）合成研究对高质量、大规模且易于处理的训练数据需求日益迫切。Emolia Filtered NanoCodec (FSQ) Tokens数据集由Nineninesix, Inc.于2026年构建，旨在解决传统语音数据集中音频质量参差不齐与原始波形存储开销巨大的双重瓶颈。该数据集以LAION发布的Emolia情感增强多语种语音数据集为源头，通过引入开源音频质量过滤工具audio_filter与NVIDIA的NanoCodec（FSQ）神经编解码模型，对原始语料进行了系统性净化与离散化编码。其核心价值在于将原始波形转化为8层紧凑的整数码流，仅保留文本、说话人、语言与编码标记，在显著降低存储与计算成本的同时，为大规模TTS模型训练提供了可直接使用的、高品质的离散音频表征，对推动语音合成领域的工程化与多语言研究具有重要影响。

当前挑战

该数据集所应对的领域核心挑战在于，原始语音数据中普遍存在的噪声、削波、带宽限制及多说话人交叠等污染因素会严重劣化TTS系统的自然度与清晰度，而传统手工清洗难以规模化。此外，高保真语音波形的高存储与传输成本成为构建大规模训练集的显著障碍。在构建过程中，团队首先面对的是设计一套鲁棒的自动质量评估流水线，需兼顾窄带与宽带音频的不同声学特性，并高效识别重叠语音。其次，采用FSQ神经编解码器将22kHz波形以低至1.89kbps的码率压缩为21.5帧/秒的离散标记，需确保重构音频质量无损，并维持8层码书间的帧级严格对齐，这对编码框架的稳定性与计算效率提出了极高要求，最终以68.9GB存储近540万条经精确对齐的文本-标记样本对的形式呈现了这一挑战的解决方案。

常用场景

经典使用场景

在语音合成与多模态生成领域，该数据集作为高质量、预编码的离散语音表征资源，被广泛用于训练端到端文本到语音系统。通过将多语种情感语音转化为八层有限标量量化（FSQ）令牌序列，研究人员可直接利用文本、说话人身份及语言标签联合建模声学特征，规避了传统波形存储冗余与加载瓶颈。尤其适用于基于神经编解码器的生成式架构，如VALL-E、SpeechGPT等离散令牌驱动模型，大幅简化数据预处理流程并提升训练效率。

实际应用

在实际产业落地中，该数据集可赋能高保真多语种语音助手、情感化有声读物生成及实时语音转换系统。基于其预编码令牌结构，开发者能快速部署轻量级神经编解码器解码模块，在智能手机、物联网设备等边缘端实现低延迟语音合成。流式加载设计支持百亿级样本的分布式训练，适用于智能客服、教育平台及无障碍阅读等场景，兼顾存储效率与合成品质。

衍生相关工作

该数据集衍生了一系列语音表示学习与生成的前沿工作，包括基于残差向量量化的语音Tokenizer优化、跨语言零样本情感迁移模型，以及结合大语言模型的语音理解-生成统一框架。其过滤后的小型化子集常被用于评估编解码器在噪声鲁棒性与音色保留间的权衡，而完整版本则支撑了如EmoVoice、MultiVerse等情感感知TTS模型的预训练，推动了离散语音令牌在自监督学习与多模态对话系统中的标准化应用。