Archit00/bbench-dep-song-describer

<div align="center"> # The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation [](https://opensource.org/licenses/MIT) [](https://arxiv.org/abs/2311.10057) [](https://doi.org/10.5281/zenodo.10072001) [](https://huggingface.co/datasets/mulab-mir/song-describer) [Ilaria Manco](https://ilariamanco.com/)\*^1,2, [Benno Weck](https://www.upf.edu/web/mtg/about/team-members/-/asset_publisher/l2XuyhfmWvQ5/content/weck-benno/maximized)\*³, [Seungheon Doh](https://seungheondoh.github.io/)⁴, [Minz Won](https://minzwon.github.io/)⁵, [Yixiao Zhang](http://www.eecs.qmul.ac.uk/~yz007/)¹, [Dmitry Bogdanov](https://dbogdanov.com/)³, [Yusong Wu](https://lukewys.github.io/)⁶, [Ke Chen](https://www.knutchen.com/)⁷, [Philip Tovstogan](https://philtgun.me/)³, [Emmanouil Benetos](http://www.eecs.qmul.ac.uk/~emmanouilb/)¹, [Elio Quinton](https://scholar.google.com/citations?user=IaciybgAAAAJ)², [George Fazekas](http://www.eecs.qmul.ac.uk/~gyorgyf/about.html)¹, [Juhan Nam](https://mac.kaist.ac.kr/~juhan/)⁴<br> ¹ QMUL, ² UMG, ³ UPF, ⁴ KAIST, ⁵ ByteDance, ⁶ MILA, ⁷ UCSD </div> * equal contribution This repository contains starter code for the Song Describer Dataset (SDD). * [Paper](https://arxiv.org/abs/2311.10057) (accepted to the [ML for Audio workshop](https://mlforaudioworkshop.com/) @ NeurIPS 2023) * [Zenodo](https://doi.org/10.5281/zenodo.10072001) * [Datasheet](docs/datasheet.md) * [Hugging Face]() ([HF demo](https://huggingface.co/spaces/renumics/spotlight-song-describer-demo) built by [@Renumics](https://github.com/Renumics)) ## Dataset overview <div align="center"> https://github.com/mulab-mir/song-describer-dataset/assets/13520622/347133af-dac0-4d40-92b6-c2fae6742927 *"A retro-futurist drum machine groove drenched in bubbly synthetic sound effects and a hint of an acid bassline."* *"Elegant and sophisticated Latin jazz piece with a Cuban base and a whispered melodic female voice."* *"Calm sitar and Indian tabla with dramatic synthetic strings background."* </div> SDD contains ~1.1k captions for 706 permissively licensed music recordings. It is designed for use in evaluation of models that address music-and-language (M&L) tasks such as music captioning, text-to-music generation and music-language retrieval. More information about the data, collection method and validation is provided in the [data card](docs/datacard.md), together with more in-depth documentation in the [datasheet](docs/datasheet.md). | Subset | Tracks | Captions | Annotators | Cap len (avg) | Vocab size | Audio len | |:----:|:----:|:-----:|:-----:|:----:|:-----:|:-----:| | full | 706 | 1106 | 142 | 21.7 ± 12.4 | 2859 | ~ 2 min | | valid| 546 | 746 | 114 | 18.2 ± 7.6 | 1942 | ~ 2 min | ## Downloading the dataset The dataset is available to download from [Zenodo](https://doi.org/10.5281/zenodo.10072001): ```bash wget -P data https://zenodo.org/record/10072001/files/song_describer.csv https://zenodo.org/record/10072001/files/audio.zip unzip data/audio.zip -d data/audio ``` A download script is also available [here](scripts/download.sh). ## Code setup To use this code, we recommend creating a new python3 virtual environment: ```bash python -m venv venv source venv/bin/activate ``` Then, clone the repository and install the dependencies: ```bash git clone https://github.com/mulab-mir/song-describer-dataset.git cd song-describer-dataset pip install -r requirements.txt ``` ## Reproducing the analysis in the paper The overview statistics presented in the paper can be reproduced via the code in the [`dataset_stats.ipynb`](exploratory_analysis/dataset_stats.ipynb) notebook. Further exploratory analysis of the data can be found in the [`data_exploration.ipynb`](exploratory_analysis/data_exploration.ipynb) notebook ## Using the dataset ### PyTorch [Coming soon] ### Hugging Face [coming soon] ### Benchmarking M&L models with SDD [coming soon] ## Cite If you use the dataset or the code in this repo, please consider citing our work: ```bib @inproceedings{manco2023thesong, title={The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation}, author={Manco, Ilaria and Weck, Benno and Doh, Seungheon and Won, Minz and Zhang, Yixiao and Bogdanov, Dmitry and Wu, Yusong and Chen, Ke and Tovstogan, Philip and Benetos, Emmanouil and Quinton, Elio and Fazekas, György and Nam, Juhan}, booktitle={Machine Learning for Audio Workshop at NeurIPS 2023}, year={2023}, } ``` ## License This repository is released under the MIT License. Please see the [LICENSE](LICENSE) file for more details. The dataset is released under the [CC BY-SA 4.0 license](https://creativecommons.org/licenses/by-sa/4.0/). ## Contact If you have any questions, please get in touch: [[e-mail]]([e-mail]).SDD contains ~1.1k captions for 706 permissively licensed music recordings. It is designed for use in evaluation of models that address music-and-language (M&L) tasks such as music captioning, text-to-music generation and music-language retrieval. Subsets include full (706 tracks, 1106 captions, 142 annotators, average caption length 21.7 ± 12.4, vocabulary size 2859, audio length ~2 min) and valid (546 tracks, 746 captions, 114 annotators, average caption length 18.2 ± 7.6, vocabulary size 1942, audio length ~2 min). Example captions: A retro-futurist drum machine groove drenched in bubbly synthetic sound effects and a hint of an acid bassline., Elegant and sophisticated Latin jazz piece with a Cuban base and a whispered melodic female voice., Calm sitar and Indian tabla with dramatic synthetic strings background.

搜集汇总

数据集介绍

构建方式

在音乐与语言跨模态研究日益兴起的背景下，该数据集通过众包方式对706段许可宽松的音乐录音进行人工标注，最终汇聚142位标注者贡献的约1.1千条自然语言描述。每段音频时长约两分钟，标注者被要求以自由文本形式描绘其感知到的音乐内容，涵盖乐器、风格、情绪及制作特征等维度。经过质量筛选与验证，形成包含完整集与验证子集的结构化语料，为音乐与语言任务的模型评估提供了可靠的基准数据。

特点

该数据集的核心特质在于其双重属性：既提供音频信号，又配以开放式的文本描述，从而支持音乐描述生成、文本到音乐生成以及音乐-语言检索等多类任务。标注文本平均长度约21.7词，词汇量达2859，展现出丰富的语言多样性。数据规模适中且许可宽松，便于学术研究与可重复实验。验证子集进一步保障了评估的严谨性，使该数据集成为音乐与语言交叉领域稀缺的公开评测资源。

使用方法

使用者可从Zenodo平台下载CSV标注文件与音频压缩包，按脚本解压后即可获得完整的音频与文本对。通过创建Python虚拟环境并安装依赖，研究者可运行仓库中的分析笔记以复现论文统计结果。虽然PyTorch与Hugging Face的集成接口尚在开发中，但当前数据格式已支持自定义加载与基准测试。引用时需遵循相应学术规范，并遵守CC BY-SA 4.0许可协议。

背景与挑战

背景概述

音乐与语言跨模态研究长期受限于缺乏高质量、可公开获取的音频描述资源。2023年，Manco、Weck等学者联合QMUL、UPF、KAIST等机构发布Song Describer数据集，收录706段许可音乐及约1.1k条自然语言标注，旨在为音乐描述生成、文本到音乐生成和音乐-语言检索等任务提供标准化评估基准。该数据集填补了音乐领域细粒度语义对齐资源的空白，推动了跨模态表征学习与生成模型的可复现评测，对音乐信息检索与自然语言处理的交叉研究具有重要影响力。

当前挑战

该数据集所应对的核心挑战在于音乐语义的主观性与语言描述的多样性：同一乐曲可能引发截然不同的情感联想与风格指涉，导致标注一致性难以保证。构建过程中，需在版权合规与音频质量之间取得平衡，并设计有效的众包协议以引导非专家标注者生成准确且富有表现力的描述。此外，如何界定描述粒度、控制词汇偏差以及确保评估指标对生成文本的鲁棒性，均为该领域尚未充分解决的难题，直接影响模型泛化能力的客观衡量。

常用场景

经典使用场景

在音乐与语言跨模态研究领域，音乐描述生成、文本到音乐生成及音乐-语言检索等任务长期受限于高质量配对数据的稀缺。该数据集凭借706段许可音乐与约1.1k条人工标注描述，为模型评估提供了标准化基准。其经典用法在于：以音乐音频为输入，检验模型生成自然语言描述的准确性；或反向以文本为条件，评估生成音乐与描述内容的语义一致性；亦可用于跨模态检索，衡量音频与文本嵌入的匹配程度。

解决学术问题

音乐与语言交叉研究长期面临公开评测资源匮乏的困境，既有数据多受版权制约或规模有限，难以支撑模型泛化能力的客观比较。该数据集通过严格的数据采集与验证流程，构建了可自由使用的音乐-文本配对语料，从而解决了音乐描述生成与文本到音乐生成任务中缺乏统一评测基准的问题。其意义在于推动可重复研究，为不同模型提供公平比较的平台，并促进跨模态表征学习方法的迭代与优化。

衍生相关工作

自发布以来，该数据集已被多项音乐与语言跨模态研究采纳为评测基准，催生了音乐描述生成、文本到音乐生成及跨模态检索等方向的对比实验与模型改进工作。相关研究以此为基础探索音频表征与语言表征的对齐策略，并衍生出面向音乐描述质量的评价指标与人工评估协议。该数据集亦激发了针对音乐语义理解的少样本与零样本学习研究，推动了音乐信息检索与自然语言处理交叉领域的持续发展。