SONYC-FSD-SED

<strong>Created by</strong> Yu Wang, Mark Cartwright, and Juan Pablo Bello <strong>Publication</strong> If using this data in academic work, please cite the following paper, which presented this dataset: Y. Wang, M. Cartwright, and J. P. Bello. "Active Few-Shot Learning for Sound Event Detection", INTERSPEECH, 2022 <strong>Description</strong> SONYC-FSD-SED is an open dataset of programmatically mixed audio clips that simulates audio data in an environmental sound monitoring system, where sound class occurrences and co-occurrences exhibit seasonal periodic patterns. We use recordings collected from the Sound of New York City (SONYC) acoustic sensor network as backgrounds, and single-labeled clips in the FSD50K dataset as foreground events to generate 576,591 10-second strongly-labeled soundscapes with Scaper (including 111,294 additional test data for the experiment of sampling window). Instead of sampling foreground sound events uniformly, we simulate the occurrence probability of each class at different times in a year, creating more realistic temporal characteristics. <strong>Source material and annotations</strong> Due to the large size of the dataset, instead of releasing the raw audio files, we release the source material and soundscape annotations in JAMS format, which can be used to reproduce SONYC-FSD-SED using Scaper with the script in the project repository. <strong>Background material from SONYC recordings</strong> We pick a sensor from the SONYC sensor network and subsample from recordings it collected within a year (2017). We categorize these ∼550k 10-second clips into 96 bins based on timestamps, where each bin represents a unique combination of the <em>month of a year</em>, <em>day of a week </em>(weekday or weekend), and <em>time of a day</em> (divided into four 6-hour blocks). Next, we run a pre-trained urban sound event classifier over all recordings and filter out clips with active sound classes. We do not filter out footstep and bird since they appear too frequently, instead, we remove these two classes from the foreground sound material. Then from each bin, we choose the clip with the lowest sound pressure level, yielding 96 background clips. <strong>Foreground material from FSD50K</strong> We follow the same filtering process as in FSD-MIX-SED to get the subset of FSD50K with short single-labeled clips. In addition, we remove two classes, "Chirp_and_tweet" and "Walk_and_footsteps", that exist in our SONYC background recordings. This results in <strong>87 sound classes</strong>. <em>vocab.json</em> contains the list of 87 classes, each class is then labeled by its index in the list. 0-42: train, 43-56: val, 57-86: test. <strong>Occurrence probability modelling</strong> For each class, we model its occurrence probability within a year. We use von Mises probability density functions to simulate the probability distribution over different weeks in a year and hours in a day considering their cyclic characteristics: \(f(x|μ, κ) = e^{κcos(x−μ)}/2πI_0(κ)\), where \(I_0(κ)\) is the modified Bessel function of order \(0\), \(\mu\) and \(1/\kappa\) are analogous to the mean and variance in the normal distribution. We randomly sample \((\mu_{year}, \mu_{day})\) from \([-\pi, \pi]\) and \((\kappa_{year}, \kappa_{day})\) from \([0, 10]\). We also randomly assign \(p_{weekday} \in [0, 1] \), \(p_{weekend} = 1 − p_{weekday}\) to simulate the probability distribution over different days in a week. Finally, we get the probability distribution over the entire year with a 1-hour resolution. At a given timestamp, we integrate \(f_{year}\) and \(f_{day}\) over the 1-hour window and multiply them together with \(p_{weekday}\) or \(p_{weekend}\) depends on the day. To speed up the following sampling process, we scale the final probability distribution using a temperature parameter randomly sampled from \([2,3]\). <strong>Files</strong> <em>SONYC_FSD_SED.source.tar.gz</em>: 96 SONYC backgrounds and 10,158 foreground sounds in `.wav` format. The original file size is 2GB. <em>SONYC_FSD_SED.annotations.tar.gz</em>: 465,467 JAMS files. The original file size is 57GB. <em>SONYC_FSD_SED_add_test.annotations.tar.gz: </em>111,294 JAMS files for additional test data. The original file size is 14GB. <em>vocab.json:</em> 87 classes. <em>occ_prob_per_cl.pkl</em>: Occurrence probability for each foreground sound class. <strong>References</strong> [1] J. P. Bello, C. T. Silva, O. Nov, R. L. DuBois, A. Arora, J. Salamon, C. Mydlarz, and H. Doraiswamy, “SONYC: A system for monitoring, analyzing, and mitigating urban noise pollution,” Commun. ACM, 2019 [2] E. Fonseca, X. Favory, J. Pons, F. Font, X. Serra. "FSD50K: an Open Dataset of Human-Labeled Sound Events", arXiv:2010.00475, 2020.**数据集制作**：由王宇（Yu Wang）、马克·卡特赖特（Mark Cartwright）与胡安·巴勃罗·贝略（Juan Pablo Bello）完成。 **学术引用规范**：若将本数据集用于学术研究，请引用下述发表于2022年国际语音通信大会（INTERSPEECH）的论文，该论文首次发布了此数据集：Y. Wang, M. Cartwright, and J. P. Bello. 《主动少样本学习用于声音事件检测（Active Few-Shot Learning for Sound Event Detection）》, INTERSPEECH, 2022。 **数据集概述**：SONYC-FSD-SED是一个开源的程序化混音音频片段数据集，旨在模拟环境声音监测系统中的音频数据，其中声音类别的出现与共现规律呈现季节性周期特征。本数据集以纽约声景（Sound of New York City, SONYC）声学传感器网络采集的录音作为背景音频，以FSD50K数据集（FSD50K）中的单标签片段作为前景事件，通过Scaper工具（Scaper）生成了576591段10秒的强标注（strongly-labeled）声景数据（其中包含111294段额外的测试数据，用于采样窗口实验）。与均匀采样前景声音事件的做法不同，我们模拟了每一类声音在一年中不同时段的出现概率，从而构建更符合真实场景的时间特性。 **源素材与标注文件**：鉴于本数据集体量较大，我们未发布原始音频文件，而是以JAMS格式（JAMS）发布了源素材与声景标注文件，使用者可通过项目仓库中的脚本结合Scaper工具复现SONYC-FSD-SED数据集。 **SONYC录制背景素材**：我们从SONYC传感器网络中选取一台传感器，提取其2017年一整年采集的录音并进行二次采样。我们将约55万段10秒的音频片段按照时间戳划分为96个区间，每个区间对应**一年中的月份**、**一周中的日期（工作日/周末）**与**一日中的时段（划分为四个6小时块）**的唯一组合。随后，我们将预训练的城市声音事件分类器（Sound Event Detection）应用于所有录音，并过滤掉包含活跃声音类别的片段；但由于脚步声与鸟鸣出现频率过高，我们未对这两类进行过滤，而是将其从前景声音素材中移除。最后，我们从每个区间中选取声压级最低的音频片段，最终得到96段背景音频。 **FSD50K前景素材**：我们沿用FSD-MIX-SED的过滤流程，从FSD50K数据集（FSD50K）中筛选出短时长单标签片段子集。此外，我们移除了SONYC背景录音中存在的"Chirp_and_tweet（鸣啼与啭鸣）"与"Walk_and_footsteps（行走与脚步声）"两个类别，最终得到87个声音类别。vocab.json文件包含这87个类别的列表，每个类别将按照其在列表中的索引进行标注：0-42为训练集，43-56为验证集，57-86为测试集。 **出现概率建模**：我们为每个类别建模其在一年内的出现概率。考虑到时间的周期性特征，我们使用冯·米塞斯（von Mises）概率密度函数来模拟一年中不同周数与一日中不同小时的概率分布，其公式为：\(f(x|μ, κ) = e^{κcos(x−μ)}/2πI_0(κ)\)，其中\(I_0(κ)\)为0阶修正贝塞尔函数（modified Bessel function），\(mu\)与\(1/kappa\)分别类似于正态分布中的均值与方差。我们从\([-π, π]\)中随机采样\((mu_{year}, mu_{day})\)，并从\([0, 10]\)中随机采样\((kappa_{year}, kappa_{day})\)。此外，我们随机为\(p_{weekday}\)赋值于\([0, 1]\)区间，令\(p_{weekend} = 1 - p_{weekday}\)，以此模拟一周内不同日期的概率分布。最终，我们得到分辨率为1小时的全年概率分布。在给定时间戳下，我们将\(f_{year}\)与\(f_{day}\)在1小时窗口内积分，再结合对应日期的\(p_{weekday}\)或\(p_{weekend}\)相乘。为加速后续采样流程，我们使用从\([2,3]\)中随机采样得到的温度参数对最终概率分布进行缩放。 **数据集文件说明**： 1. SONYC_FSD_SED.source.tar.gz：包含96段SONYC背景音频与10158段前景音频，格式为.wav，原始文件大小为2GB。 2. SONYC_FSD_SED.annotations.tar.gz：包含465467个JAMS格式（JAMS）标注文件，原始文件大小为57GB。 3. SONYC_FSD_SED_add_test.annotations.tar.gz：包含111294个用于额外测试数据的JAMS格式（JAMS）标注文件，原始文件大小为14GB。 4. vocab.json：包含87个声音类别的列表文件。 5. occ_prob_per_cl.pkl：存储每个前景声音类别的出现概率文件。 **参考文献**： [1] J. P. Bello, C. T. Silva, O. Nov, R. L. DuBois, A. Arora, J. Salamon, C. Mydlarz, and H. Doraiswamy, "SONYC: A system for monitoring, analyzing, and mitigating urban noise pollution", 《ACM通讯（Commun. ACM）》, 2019 [2] E. Fonseca, X. Favory, J. Pons, F. Font, X. Serra. 《FSD50K：人类标注声音事件开源数据集（FSD50K: an Open Dataset of Human-Labeled Sound Events）》, arXiv:2010.00475, 2020。