ajd12342/paraspeechclap-eval-combined

--- language: - en license: cc-by-nc-sa-4.0 tags: - speech - audio - style - CLAP - dual-encoder - evaluation - benchmark - compositional - intrinsic - situational source_datasets: - ajd12342/paraspeechcaps task_categories: - audio-classification size_categories: - 1K<n<10K dataset_info: features: - name: source dtype: string - name: relative_audio_path dtype: string - name: text_description dtype: string - name: transcription dtype: string - name: intrinsic_tags sequence: string - name: situational_tags dtype: string - name: basic_tags sequence: string - name: all_tags sequence: string - name: speakerid dtype: string - name: name dtype: string - name: duration dtype: float64 - name: gender dtype: string - name: accent dtype: string - name: pitch dtype: string - name: speaking_rate dtype: string - name: noise dtype: string - name: utterance_pitch_mean dtype: float32 - name: snr dtype: float64 - name: phonemes dtype: string splits: - name: test num_bytes: 1326439 num_examples: 1432 download_size: 352784 dataset_size: 1326439 configs: - config_name: default data_files: - split: test path: data/test-* --- # Combined Evaluation Dataset for ParaSpeechCLAP Models Evaluation dataset for **combined (compositional)** style attributes, used in the paper: *ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining* Anuj Diwan, Eunsol Choi, David Harwath ## Overview This dataset is the **combined evaluation set** for the ParaSpeechCLAP model family. It contains speech clips paired with **compositional style captions that include both intrinsic and situational tags**, drawn from the **Expresso-EARS** portion of the [ParaSpeechCaps](https://huggingface.co/datasets/ajd12342/paraspeechcaps) holdout set. It is used to perform **retrieval** evaluation. This version uses the **original captions** containing both intrinsic and situational tags from the [ParaSpeechCaps](https://huggingface.co/datasets/ajd12342/paraspeechcaps) holdout set. ## Installation Install the `datasets` package to load the dataset: ```bash pip install datasets ``` To run retrieval evaluation with ParaSpeechCLAP, install the [ParaSpeechCLAP GitHub repository](https://github.com/ajd12342/paraspeechclap): ```bash git clone https://github.com/ajd12342/paraspeechclap.git cd paraspeechclap pip install -r requirements.txt ``` ### Setting up audio files The dataset contains a `relative_audio_path` column but not the audio files themselves. Resolving audio paths requires specifying `data.audio_root`, a common root directory organized as `${audio_root}/{source}/`, where `{source}` matches the value of the `source` column in the dataset. This dataset uses the **Expresso** and **EARS** sources. Follow the [ParaSpeechCaps audio setup instructions](https://github.com/ajd12342/paraspeechcaps/tree/main/dataset#22-processing-dataset-audio) for those sources, with the following adjustment: instead of placing each source at its own root directory, place them under a common root: - `${audio_root}/expresso/` (instead of `${expresso_root}`) - `${audio_root}/ears/` (instead of `${ears_root}`) Then pass `data.audio_root=${audio_root}` when running any ParaSpeechCLAP script. ## Usage with ParaSpeechCLAP ### Retrieval evaluation ```bash python scripts/evaluate_retrieval.py \ --config-name eval/retrieval \ checkpoint_path=./checkpoints/paraspeechclap-combined.pth.tar \ data.dataset_name=ajd12342/paraspeechclap-eval-combined \ data.audio_root=/path/to/audio_root \ meta.results=./results_retrieval/paraspeechclap-eval-combined/ajd12342-paraspeechclap-combined ``` ### Loading the dataset ```python from datasets import load_dataset dataset = load_dataset("ajd12342/paraspeechclap-eval-combined", split="test") print(f"Number of clips: {len(dataset)}") print(f"Number of unique prompts: {len(set(dataset['text_description']))}") print(dataset[0]) ``` ## Related Resources - **GitHub Repository:** [https://github.com/ajd12342/paraspeechclap](https://github.com/ajd12342/paraspeechclap) - **Models:** [ajd12342/paraspeechclap-intrinsic](https://huggingface.co/ajd12342/paraspeechclap-intrinsic), [ajd12342/paraspeechclap-situational](https://huggingface.co/ajd12342/paraspeechclap-situational) and [ajd12342/paraspeechclap-combined](https://huggingface.co/ajd12342/paraspeechclap-combined) - **Parent Dataset:** [https://huggingface.co/datasets/ajd12342/paraspeechcaps](https://huggingface.co/datasets/ajd12342/paraspeechcaps) - **Training Datasets:** [https://huggingface.co/datasets/ajd12342/paraspeechcaps-intrinsic-train](https://huggingface.co/datasets/ajd12342/paraspeechcaps-intrinsic-train) and [https://huggingface.co/datasets/ajd12342/paraspeechcaps-situational-train](https://huggingface.co/datasets/ajd12342/paraspeechcaps-situational-train) - **Evaluation Datasets:** [https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-intrinsic](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-intrinsic), [https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-situational](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-situational) and [https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-combined](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-combined) ## Citation ```bibtex @inproceedings{diwan2026paraspeechclap, title={{ParaSpeechCLAP}: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining}, author={Diwan, Anuj and Choi, Eunsol and Harwath, David}, journal={Under Review}, year={2026} } ```语言： - 英语 许可协议：CC BY-NC-SA 4.0 标签： - 语音 - 音频 - 风格 - CLAP - 双编码器（dual-encoder） - 评估 - 基准测试 - 组合式 - 内在属性 - 情境属性 源数据集： - ajd12342/paraspeechcaps 任务类别： - 音频分类 数据规模：1K<n<10K 数据集信息： 特征： - 名称：source，数据类型：字符串 - 名称：relative_audio_path，数据类型：字符串 - 名称：text_description，数据类型：字符串 - 名称：transcription，数据类型：字符串 - 名称：intrinsic_tags，数据类型：字符串序列 - 名称：situational_tags，数据类型：字符串 - 名称：basic_tags，数据类型：字符串序列 - 名称：all_tags，数据类型：字符串序列 - 名称：speakerid，数据类型：字符串 - 名称：name，数据类型：字符串 - 名称：duration，数据类型：float64 - 名称：gender，数据类型：字符串 - 名称：accent，数据类型：字符串 - 名称：pitch，数据类型：字符串 - 名称：speaking_rate，数据类型：字符串 - 名称：noise，数据类型：字符串 - 名称：utterance_pitch_mean，数据类型：float32 - 名称：snr，数据类型：float64 - 名称：phonemes，数据类型：字符串 数据划分： - 名称：test（测试集），字节数：1326439，样本数：1432 下载大小：352784，数据集总大小：1326439 配置项： - 配置名称：default（默认配置），数据文件： - 划分：test，路径：data/test-* # 适用于ParaSpeechCLAP模型的组合式评估数据集 **组合式风格属性**评估数据集，用于论文： *《ParaSpeechCLAP：面向富风格语言-音频预训练的双编码器语音-文本模型》* 作者：Anuj Diwan、Eunsol Choi、David Harwath ## 概述 本数据集为ParaSpeechCLAP模型家族的**组合式评估集**，包含语音片段与**兼具内在属性与情境属性标签的组合式风格描述文本**，数据取自源数据集[ParaSpeechCaps](https://huggingface.co/datasets/ajd12342/paraspeechcaps)预留测试集的**Expresso-EARS**子集，用于执行**检索任务**评估。本版本采用源自ParaSpeechCaps预留测试集的、同时包含内在与情境属性标签的**原始描述文本**。 ## 安装 通过安装`datasets`库加载本数据集： bash pip install datasets 若需使用ParaSpeechCLAP执行检索评估，请安装[ParaSpeechCLAP GitHub仓库](https://github.com/ajd12342/paraspeechclap)： bash git clone https://github.com/ajd12342/paraspeechclap.git cd paraspeechclap pip install -r requirements.txt ### 音频文件配置 本数据集包含`relative_audio_path`（相对音频路径）列，但未附带音频文件本身。需通过指定`data.audio_root`参数配置音频根目录，目录组织格式为`${audio_root}/{source}/`，其中`{source}`与数据集中`source`列的取值一一对应。 本数据集使用**Expresso**与**EARS**两个数据源，请参照[ParaSpeechCaps音频配置指南](https://github.com/ajd12342/paraspeechcaps/tree/main/dataset#22-processing-dataset-audio)进行设置，仅需调整如下：无需将每个数据源置于独立根目录，而是将二者统一置于公共根目录下： - `${audio_root}/expresso/`（替代原`${expresso_root}`） - `${audio_root}/ears/`（替代原`${ears_root}`） 在运行ParaSpeechCLAP相关脚本时，请传入参数`data.audio_root=${audio_root}`。 ## 与ParaSpeechCLAP配合使用 ### 检索评估 bash python scripts/evaluate_retrieval.py --config-name eval/retrieval checkpoint_path=./checkpoints/paraspeechclap-combined.pth.tar data.dataset_name=ajd12342/paraspeechclap-eval-combined data.audio_root=/path/to/audio_root meta.results=./results_retrieval/paraspeechclap-eval-combined/ajd12342-paraspeechclap-combined ### 加载数据集 python from datasets import load_dataset dataset = load_dataset("ajd12342/paraspeechclap-eval-combined", split="test") print(f"语音片段总数：{len(dataset)}") print(f"唯一提示文本数量：{len(set(dataset['text_description']))}") print(dataset[0]) ## 相关资源 - **GitHub仓库**：[https://github.com/ajd12342/paraspeechclap](https://github.com/ajd12342/paraspeechclap) - **模型权重**：[ajd12342/paraspeechclap-intrinsic](https://huggingface.co/ajd12342/paraspeechclap-intrinsic)、[ajd12342/paraspeechclap-situational](https://huggingface.co/ajd12342/paraspeechclap-situational) 与 [ajd12342/paraspeechclap-combined](https://huggingface.co/ajd12342/paraspeechclap-combined) - **父级数据集**：[https://huggingface.co/datasets/ajd12342/paraspeechcaps](https://huggingface.co/datasets/ajd12342/paraspeechcaps) - **训练数据集**：[ajd12342/paraspeechcaps-intrinsic-train](https://huggingface.co/datasets/ajd12342/paraspeechcaps-intrinsic-train) 与 [ajd12342/paraspeechcaps-situational-train](https://huggingface.co/datasets/ajd12342/paraspeechcaps-situational-train) - **评估数据集**：[ajd12342/paraspeechclap-eval-intrinsic](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-intrinsic)、[ajd12342/paraspeechclap-eval-situational](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-situational) 与 [ajd12342/paraspeechclap-eval-combined](https://huggingface.co/datasets/ajd12342/paraspeechclap-eval-combined) ## 引用 bibtex @inproceedings{diwan2026paraspeechclap, title={{ParaSpeechCLAP}: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining}, author={Diwan, Anuj and Choi, Eunsol and Harwath, David}, journal={Under Review}, year={2026} }