ajd12342/paraspeechcaps-intrinsic-train

--- language: - en license: cc-by-nc-sa-4.0 tags: - speech - audio - style - CLAP - dual-encoder - contrastive-learning - intrinsic - speaker-level source_datasets: - ajd12342/paraspeechcaps task_categories: - audio-classification size_categories: - 100K<n<1M dataset_info: features: - name: source dtype: string - name: relative_audio_path dtype: string - name: text_description sequence: string - name: transcription dtype: string - name: intrinsic_tags sequence: string - name: situational_tags sequence: string - name: basic_tags sequence: string - name: all_tags sequence: string - name: speakerid dtype: string - name: name dtype: string - name: duration dtype: float64 - name: gender dtype: string - name: accent dtype: string - name: pitch dtype: string - name: speaking_rate dtype: string - name: noise dtype: string - name: utterance_pitch_mean dtype: float64 - name: snr dtype: float64 - name: phonemes dtype: string splits: - name: train num_bytes: 925936580 num_examples: 944820 download_size: 321934506 dataset_size: 925936580 configs: - config_name: default data_files: - split: train path: data/train-* --- # ParaSpeechCaps Intrinsic Training Dataset Training dataset for the **ParaSpeechCLAP-Intrinsic** and **ParaSpeechCLAP-Combined** models, from the paper: [*ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining*](https://arxiv.org/abs/2603.28737) Anuj Diwan, Eunsol Choi, David Harwath *Under review* This dataset contains the **intrinsic-tag subset** of [ParaSpeechCaps](https://huggingface.co/datasets/ajd12342/paraspeechcaps), filtered to include examples annotated with at least one **intrinsic (speaker-level)** style tag. It is used to train the ParaSpeechCLAP-Intrinsic model with a contrastive + classification multitask loss and class-balanced sampling and the ParaSpeechCLAP-Combined model with a contrastive loss. ## Installation Install the `datasets` package to load the dataset: ```bash pip install datasets ``` To train ParaSpeechCLAP models using this dataset, install the [ParaSpeechCLAP GitHub repository](https://github.com/ajd12342/paraspeechclap): ```bash git clone https://github.com/ajd12342/paraspeechclap.git cd paraspeechclap pip install -r requirements.txt ``` ### Setting up audio files The dataset contains a `relative_audio_path` column but not the audio files themselves. Resolving audio paths requires specifying `data.audio_root`, a common root directory organized as `${audio_root}/{source}/`, where `{source}` matches the value of the `source` column in the dataset. This dataset includes examples from **VoxCeleb**, **Expresso**, **EARS**, and **Emilia**. Follow the [ParaSpeechCaps audio setup instructions](https://github.com/ajd12342/paraspeechcaps/tree/main/dataset#22-processing-dataset-audio) for those sources, with the following adjustment: instead of placing each source at its own root directory, place them under a common root: - `${audio_root}/voxceleb/` (instead of `${voxceleb_root}`) - `${audio_root}/expresso/` (instead of `${expresso_root}`) - `${audio_root}/ears/` (instead of `${ears_root}`) - `${audio_root}/emilia/` (instead of `${emilia_root}`) Then pass `data.audio_root=${audio_root}` when running any ParaSpeechCLAP script. ## Usage with ParaSpeechCLAP ### Training ```bash torchrun --nproc_per_node=4 scripts/train.py \ --config-name train/intrinsic \ data.audio_root=/path/to/audio_root \ meta.results=./experiments ``` ### Loading the dataset ```python from datasets import load_dataset dataset = load_dataset("ajd12342/paraspeechcaps-intrinsic-train", split="train") print(f"Number of examples: {len(dataset)}") print(dataset[0]) ``` ## Related Resources - **GitHub Repository:** [https://github.com/ajd12342/paraspeechclap](https://github.com/ajd12342/paraspeechclap) - **Models:** [ajd12342/paraspeechclap-intrinsic](https://huggingface.co/ajd12342/paraspeechclap-intrinsic), [ajd12342/paraspeechclap-situational](https://huggingface.co/ajd12342/paraspeechclap-situational) and [ajd12342/paraspeechclap-combined](https://huggingface.co/ajd12342/paraspeechclap-combined) - **Parent Dataset:** [https://huggingface.co/datasets/ajd12342/paraspeechcaps](https://huggingface.co/datasets/ajd12342/paraspeechcaps) - **Training Datasets:** [https://huggingface.co/datasets/ajd12342/paraspeechcaps-intrinsic-train](https://huggingface.co/datasets/ajd12342/paraspeechcaps-intrinsic-train) and [https://huggingface.co/datasets/ajd12342/paraspeechcaps-situational-train](https://huggingface.co/datasets/ajd12342/paraspeechcaps-situational-train) - **Evaluation Datasets:** [https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-intrinsic](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-intrinsic), [https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-situational](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-situational) and [https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-combined](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-combined) ## Citation ```bibtex @misc{diwan2026paraspeechclapdualencoderspeechtextmodel, title={ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining}, author={Anuj Diwan and Eunsol Choi and David Harwath}, year={2026}, eprint={2603.28737}, archivePrefix={arXiv}, primaryClass={eess.AS}, url={https://arxiv.org/abs/2603.28737}, } ```语言： - en 许可协议：cc-by-nc-sa-4.0 标签： - 语音 - 音频 - 风格 - CLAP - 双编码器（dual-encoder） - 对比学习（contrastive-learning） - 内在属性（intrinsic） - 说话人级（speaker-level） 源数据集： - ajd12342/paraspeechcaps 任务类别： - 音频分类 样本规模： - 100K<n<1M 数据集信息： 特征： - 名称：source，数据类型：字符串 - 名称：relative_audio_path，数据类型：字符串 - 名称：text_description，数据类型：字符串序列 - 名称：transcription，数据类型：字符串 - 名称：intrinsic_tags，数据类型：字符串序列 - 名称：situational_tags，数据类型：字符串序列 - 名称：basic_tags，数据类型：字符串序列 - 名称：all_tags，数据类型：字符串序列 - 名称：speakerid，数据类型：字符串 - 名称：name，数据类型：字符串 - 名称：duration，数据类型：float64 - 名称：gender，数据类型：字符串 - 名称：accent，数据类型：字符串 - 名称：pitch，数据类型：字符串 - 名称：speaking_rate，数据类型：字符串 - 名称：noise，数据类型：字符串 - 名称：utterance_pitch_mean，数据类型：float64 - 名称：snr（信噪比），数据类型：float64 - 名称：phonemes，数据类型：字符串 划分： - 名称：train，字节数：925936580，样本数：944820 下载大小：321934506 数据集总大小：925936580 配置： - 配置名称：default 数据文件： - 划分：train 路径：data/train-* # ParaSpeechCaps 内在属性训练数据集 本数据集用于**ParaSpeechCLAP-内在属性（ParaSpeechCLAP-Intrinsic）**与**ParaSpeechCLAP-组合式（ParaSpeechCLAP-Combined）**模型的训练，源自论文： [*ParaSpeechCLAP：面向丰富风格化语言-音频预训练的双编码器语音-文本模型*](https://arxiv.org/abs/2603.28737) 作者：Anuj Diwan、Eunsol Choi、David Harwath，待刊。 本数据集是[ParaSpeechCaps](https://huggingface.co/datasets/ajd12342/paraspeechcaps)的**内在属性标注子集**，经筛选后仅保留至少标注有一项**内在属性（说话人级）**风格标签的样本。本数据集用于以对比+分类多任务损失与类别平衡采样策略训练ParaSpeechCLAP-Intrinsic模型，同时也可用于以对比损失训练ParaSpeechCLAP-Combined模型。 ## 安装 安装`datasets`库以加载本数据集： bash pip install datasets 若需使用本数据集训练ParaSpeechCLAP模型，请安装[ParaSpeechCLAP GitHub仓库](https://github.com/ajd12342/paraspeechclap)： bash git clone https://github.com/ajd12342/paraspeechclap.git cd paraspeechclap pip install -r requirements.txt ### 音频文件配置 本数据集包含`relative_audio_path`列，但未附带音频文件本身。需通过指定`data.audio_root`参数解析音频路径，该参数为通用根目录，组织格式为`${audio_root}/{source}/`，其中`{source}`与数据集中`source`列的值对应。 本数据集包含来自**VoxCeleb**、**Expresso**、**EARS**与**Emilia**的样本。请参照[ParaSpeechCaps音频配置指南](https://github.com/ajd12342/paraspeechcaps/tree/main/dataset#22-processing-dataset-audio)进行配置，仅需调整如下：无需将各数据源置于各自的根目录下，而是将其统一置于通用根目录下： - `${audio_root}/voxceleb/`（替代原`${voxceleb_root}`） - `${audio_root}/expresso/`（替代原`${expresso_root}`） - `${audio_root}/ears/`（替代原`${ears_root}`） - `${audio_root}/emilia/`（替代原`${emilia_root}`） 在运行任何ParaSpeechCLAP脚本时，请传入`data.audio_root=${audio_root}`参数。 ## 配合ParaSpeechCLAP使用 ### 训练 bash torchrun --nproc_per_node=4 scripts/train.py --config-name train/intrinsic data.audio_root=/path/to/audio_root meta.results=./experiments ### 加载数据集 python from datasets import load_dataset dataset = load_dataset("ajd12342/paraspeechcaps-intrinsic-train", split="train") print(f"样本总数：{len(dataset)}") print(dataset[0]) ## 相关资源 - **GitHub仓库**：[https://github.com/ajd12342/paraspeechclap](https://github.com/ajd12342/paraspeechclap) - **模型**：[ajd12342/paraspeechclap-intrinsic](https://huggingface.co/ajd12342/paraspeechclap-intrinsic)、[ajd12342/paraspeechclap-situational](https://huggingface.co/ajd12342/paraspeechclap-situational)与[ajd12342/paraspeechclap-combined](https://huggingface.co/ajd12342/paraspeechclap-combined) - **父数据集**：[https://huggingface.co/datasets/ajd12342/paraspeechcaps](https://huggingface.co/datasets/ajd12342/paraspeechcaps) - **训练数据集**：[https://huggingface.co/datasets/ajd12342/paraspeechcaps-intrinsic-train](https://huggingface.co/datasets/ajd12342/paraspeechcaps-intrinsic-train)与[https://huggingface.co/datasets/ajd12342/paraspeechcaps-situational-train](https://huggingface.co/datasets/ajd12342/paraspeechcaps-situational-train) - **评估数据集**：[https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-intrinsic](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-intrinsic)、[https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-situational](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-situational)与[https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-combined](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-combined) ## 引用 bibtex @misc{diwan2026paraspeechclapdualencoderspeechtextmodel, title={ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining}, author={Anuj Diwan and Eunsol Choi and David Harwath}, year={2026}, eprint={2603.28737}, archivePrefix={arXiv}, primaryClass={eess.AS}, url={https://arxiv.org/abs/2603.28737}, }