treadon/speech-dac-tokens-3cb

---
language:
- en
license: cc-by-4.0
task_categories:
- text-to-speech
- audio-classification
tags:
- dac
- audio-tokens
- speech
- tts
- codebook
- descript-audio-codec
- librispeech
pretty_name: Speech DAC Tokens (3 Codebooks)
size_categories:
- 100K<n<1M
---

# Speech DAC Tokens (3 Codebooks)

Pre-tokenized speech dataset using the [Descript Audio Codec (DAC)](https://github.com/descriptinc/descript-audio-codec). Each audio clip has been encoded into discrete codebook tokens from DAC's first 3 residual vector quantization codebooks, paired with its text transcription.

## Dataset Summary

| Stat | Value |
|------|-------|
| **Total samples** | 241,451 |
| **Total audio** | ~780 hours |
| **Language** | English |
| **Codebooks** | 3 (of DAC's 9) |
| **Codebook size** | 1,024 entries each |
| **DAC model** | 44kHz |
| **Tokens per second** | ~258 (86 frames x 3 codebooks) |
| **Token sequence length** | 219-4,096 (mean: 3,063) |
| **Audio duration range** | ~0.8s-15.7s |

## Data Sources

| Source | Split | Clips | License |
|--------|-------|-------|---------|
| [LibriSpeech](https://www.openslr.org/12) clean-100 | train.100 | ~24,200 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| [LibriSpeech](https://www.openslr.org/12) clean-360 | train.360 | ~88,500 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| [LibriSpeech](https://www.openslr.org/12) other-500 | train.500 | ~128,750 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |

## Format

Each row contains:

| Column | Type | Description |
|--------|------|-------------|
| `text` | string | Original text transcription |
| `prompt` | string | Full training prompt: `{text}<\|audio_start\|><\|c1_X\|><\|c2_Y\|><\|c3_Z\|>...<\|audio_end\|>` |
| `input_ids` | list[int] | Pre-tokenized 3-codebook prompt. Ready for training. |
| `input_ids_1cb` | list[int] | Pre-tokenized 1-codebook prompt (c1 only, shorter sequences). |
| `input_ids_2cb` | list[int] | Pre-tokenized 2-codebook prompt (c1+c2). |
| `attention_mask` | list[int] | All 1s, same length as input_ids (3cb). |
| `labels` | list[int] | Copy of input_ids (3cb). Used as training targets. |
| `n_audio_frames` | int | Number of DAC time frames |
| `n_tokens` | int | Total token count (text + audio tokens) |

Audio tokens are interleaved per time frame: `c1, c2, c3, c1, c2, c3, ...` where:
- **c1** (codebook 1): Coarse structure - pitch, rhythm, broad spectral shape
- **c2** (codebook 2): Fine detail - residual from c1
- **c3** (codebook 3): Finest detail - residual from c1+c2

## Use Cases

### Text-to-Speech Training
Train a language model to predict DAC tokens from text input. The model learns to generate the audio token sequence, which is then decoded back to audio using DAC's decoder. No spectrogram or vocoder needed - just token prediction.

```
Input: Hello world
Output: <|audio_start|><|c1_551|><|c2_118|><|c3_42|>...<|audio_end|>
-> DAC decoder -> audio waveform
```

### Audio Language Modeling
Train unconditional or conditional audio generation models using discrete tokens, similar to how language models generate text.

### Speech Understanding
Use the tokenized representation for speech classification, speaker identification, or other downstream tasks that benefit from discrete audio representations.

### Codec Research
Study the information captured at different codebook levels, or compare DAC's tokenization against other codecs (EnCodec, SpeechTokenizer).

## How to Decode Audio

```python
import torch
import dac
from dac.utils import load_model
import re

# Load DAC decoder
dac_model = load_model(tag="latest", model_type="44khz")
dac_model.eval()

# Parse tokens from a prompt
prompt = dataset[0]["prompt"]
pattern = r'<\|c(\d+)_(\d+)\|>'
matches = re.findall(pattern, prompt)

# Group into frames (every 3 tokens = 1 frame)
frames = []
frame = [None, None, None]
for cb_str, val_str in matches:
cb = int(cb_str) - 1
frame[cb] = int(val_str)
if cb == 2:
frames.append(list(frame))
frame = [None, None, None]

# Decode: pad to 9 codebooks (DAC expects all 9)
codes = torch.tensor(frames).T.unsqueeze(0).long()
full_codes = torch.zeros(1, 9, codes.shape[2], dtype=torch.long)
full_codes[:, :3, :] = codes

with torch.no_grad():
z = dac_model.quantizer.from_codes(full_codes)
audio = dac_model.decode(z)

# audio[0, 0] is the waveform at 44100 Hz
```

## Related

- **Training code:** [treadon/ri-tts](https://github.com/treadon/ri-tts) on GitHub

## Processing Details

- Audio resampled from 16kHz (LibriSpeech native) to 44.1kHz (DAC native)
- Clips exceeding 4,096 tokens were excluded (~17% of source data)
- DAC encoding performed on Apple MPS (M4 Max) at ~2.4 clips/sec
- No word-level alignment or prosodic features - raw text + DAC codes only
- 0 CPU fallback failures during encoding

## Citation

If you use this dataset, please cite the original data sources:

```bibtex
@inproceedings{panayotov2015librispeech,
title={Librispeech: an ASR corpus based on public domain audio books},
author={Panayotov, Vassil and Chen, Guoguo and Povey, Daniel and Khudanpur, Sanjeev},
booktitle={ICASSP},
year={2015}
}

@article{kumar2024high,
title={High-fidelity audio compression with improved RVQGAN},
author={Kumar, Rithesh and others},
journal={NeurIPS},
year={2024}
}
```

---
语言：
- 英语
许可协议：CC BY 4.0
任务类别：
- 文本转语音
- 音频分类
标签：
- dac
- 音频Token（Audio Tokens）
- 语音
- TTS（文本转语音）
- 码本（codebook）
- descript-audio-codec
- LibriSpeech
展示名称：Speech DAC Tokens（3个码本）
样本规模：
- 10万 < 样本数 < 100万
---

# Speech DAC Tokens（3个码本）

本数据集为基于[Descript音频编解码器（Descript Audio Codec, DAC）](https://github.com/descriptinc/descript-audio-codec)构建的预Token化语音数据集。每条音频片段均已通过DAC的前3个残差矢量量化码本编码为离散的音频Token，并与其原始文本转录内容配对。

## 数据集概览

| 统计项 | 数值 |
|------|-------|
| **总样本数** | 241,451 |
| **总音频时长** | 约780小时 |
| **语言** | 英语 |
| **使用码本数** | 3个（DAC共9个码本） |
| **单码本容量** | 每个码本含1024个条目 |
| **DAC模型规格** | 44kHz |
| **每秒Token数** | 约258个（86帧 × 3个码本） |
| **Token序列长度** | 219~4096（平均3063） |
| **音频时长范围** | 约0.8秒~15.7秒 |

## 数据源

| 数据源 | 数据集划分 | 片段数量 | 许可协议 |
|--------|-------|-------|---------|
| [LibriSpeech](https://www.openslr.org/12) clean-100 | train.100 | 约24,200 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| [LibriSpeech](https://www.openslr.org/12) clean-360 | train.360 | 约88,500 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| [LibriSpeech](https://www.openslr.org/12) other-500 | train.500 | 约128,750 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |

## 数据格式

| 列名 | 数据类型 | 描述 |
|--------|------|-------------|
| `text` | 字符串 | 原始文本转录内容 |
| `prompt` | 字符串 | 完整训练提示：`{text}<|audio_start|><|c1_X|><|c2_Y|><|c3_Z|>...<|audio_end|>` |
| `input_ids` | 整数列表 | 预Token化的3码本提示，可直接用于训练 |
| `input_ids_1cb` | 整数列表 | 预Token化的单码本提示（仅使用c1，序列更短） |
| `input_ids_2cb` | 整数列表 | 预Token化的双码本提示（使用c1+c2） |
| `attention_mask` | 整数列表 | 全1张量，长度与3码本的`input_ids`一致 |
| `labels` | 整数列表 | 与3码本的`input_ids`完全一致，用作训练目标 |
| `n_audio_frames` | 整数 | DAC时间帧数 |
| `n_tokens` | 整数 | 总Token数（文本Token + 音频Token） |

音频Token按时间帧交错排列：`c1, c2, c3, c1, c2, c3, ...`，各码本含义如下：
- **c1（码本1）**：捕获粗粒度结构，包括音高、节奏与广谱形状
- **c2（码本2）**：捕获细粒度细节，为c1编码的残差
- **c3（码本3）**：捕获最精细的细节，为c1+c2编码的残差

## 应用场景

### 文本转语音训练
训练大语言模型从文本输入预测DAC Token。模型将学习生成音频Token序列，后续可通过DAC解码器将其还原为音频波形。无需使用频谱图或声码器，仅需完成Token预测任务。

输入： Hello world
输出： <|audio_start|><|c1_551|><|c2_118|><|c3_42|>...<|audio_end|>
-> DAC解码器 -> 音频波形

### 音频语言建模
使用离散Token训练无条件或条件式音频生成模型，逻辑与大语言模型生成文本一致。

### 语音理解
可将该Token化表征用于语音分类、说话人识别等可从离散音频表征中获益的下游任务。

### 编解码器研究
可用于研究不同码本层级所捕获的音频信息，或对比DAC的Token化方案与其他编解码器（如EnCodec、SpeechTokenizer）的差异。

## 音频解码方法

python
import torch
import dac
from dac.utils import load_model
import re

# 加载DAC解码器
dac_model = load_model(tag="latest", model_type="44khz")
dac_model.eval()

# 从提示中解析Token
prompt = dataset[0]["prompt"]
pattern = r'<|c(d+)_(d+)|>'
matches = re.findall(pattern, prompt)

# 按帧分组（每3个Token对应1帧）
frames = []
frame = [None, None, None]
for cb_str, val_str in matches:
cb = int(cb_str) - 1
frame[cb] = int(val_str)
if cb == 2:
frames.append(list(frame))
frame = [None, None, None]

# 解码：补全至9个码本（DAC要求输入全部9个码本）
codes = torch.tensor(frames).T.unsqueeze(0).long()
full_codes = torch.zeros(1, 9, codes.shape[2], dtype=torch.long)
full_codes[:, :3, :] = codes

with torch.no_grad():
z = dac_model.quantizer.from_codes(full_codes)
audio = dac_model.decode(z)

# audio[0, 0] 为44100 Hz采样率的音频波形

## 相关资源
- **训练代码**：GitHub上的[treadon/ri-tts](https://github.com/treadon/ri-tts)

## 数据处理细节
- 音频重采样：将LibriSpeech原生的16kHz音频重采样为DAC原生的44.1kHz
- 过滤规则：移除Token长度超过4096的片段（约占源数据的17%）
- DAC编码环境：基于Apple MPS（M4 Max）完成，编码速度约为2.4段/秒
- 未引入词级对齐或韵律特征，仅保留原始文本与DAC编码结果
- 编码过程中无CPU回退失败案例

## 引用说明
若使用本数据集，请引用以下原始数据源：

bibtex
@inproceedings{panayotov2015librispeech,
title={Librispeech: an ASR corpus based on public domain audio books},
author={Panayotov, Vassil and Chen, Guoguo and Povey, Daniel and Khudanpur, Sanjeev},
booktitle={ICASSP},
year={2015}
}

@article{kumar2024high,
title={High-fidelity audio compression with improved RVQGAN},
author={Kumar, Rithesh and others},
journal={NeurIPS},
year={2024}
}