Free Universal Sound Separation Dataset

The Free Universal Sound Separation (FUSS) Dataset is a database of arbitrary sound mixtures and source-level references, for use in experiments on arbitrary sound separation. This is the official sound separation data for the DCASE2020 Challenge Task 4: Sound Event Detection and Separation in Domestic Environments. <strong>Citation: </strong>If you use the FUSS dataset or part of it, please cite our paper describing the dataset and baseline [1]. FUSS is based on FSD data so please also cite [2]: <strong>Overview: </strong>FUSS audio data is sourced from a pre-release of Freesound dataset known as (FSD50k), a sound event dataset composed of Freesound content annotated with labels from the AudioSet Ontology. Using the FSD50K labels, these source files have been screened such that they likely only contain a single type of sound. Labels are not provided for these source files, and are not considered part of the challenge. For the purpose of the DCASE Task4 Sound Separation and Event Detection challenge, systems should not use FSD50K labels, even though they may become available upon FSD50K release. To create mixtures, 10 second clips of sources are convolved with simulated room impulse responses and added together. Each 10 second mixture contains between 1 and 4 sources. Source files longer than 10 seconds are considered "background" sources. Every mixture contains one background source, which is active for the entire duration. We provide: a software recipe to create the dataset, the room impulse responses, and the original source audio. <strong>Motivation for use in DCASE2020 Challenge Task 4: </strong> This dataset provides a platform to investigate how source separation may help with event detection and vice versa. Previous work has shown that universal sound separation (separation of arbitrary sounds) is possible [3], and that event detection can help with universal sound separation [4]. It remains to be seen whether sound separation can help with event detection. Event detection is more difficult in noisy environments, and so separation could be a useful pre-processing step. Data with strong labels for event detection are relatively scarce, especially when restricted to specific classes within a domain. In contrast, source separation data needs no event labels for training, and may be more plentiful. In this setting, the idea is to utilize larger unlabeled separation data to train separation systems, which can serve as a front-end to event-detection systems trained on more limited data. <strong>Room simulation: </strong>Room impulse responses are simulated using the image method with frequency-dependent walls. Each impulse corresponds to a rectangular room of random size with random wall materials, where a single microphone and up to 4 sources are placed at random spatial locations. <strong>Recipe for data creation: </strong>The data creation recipe starts with scripts, based on scaper [5], to generate mixtures of events with random timing of source events, along with a background source that spans the duration of the mixture clip. The scipts for this are at this GitHub repo. The data are reverberated using a different room simulation for each mixture. In this simulation each source has its own reverberation corresponding to a different spatial location. The reverberated mixtures are created by summing over the reverberated sources. The dataset recipe scripts support modification, so that participants may remix and augment the training data as desired. The constituent source files for each mixture are also generated for use as references for training and evaluation. The dataset recipe scripts support modification, so that participants may remix and augment the training data as desired. Note: no attempt was made to remove digital silence from the freesound source data, so some reference sources may include digital silence, and there are a few mixtures where the background reference is all digital silence. Digital silence can also be observed in the event recognition public evaluation data, so it is important to be able to handle this in practice. Our evaluation scripts handle it by ignoring any reference sources that are silent. <strong>Format: </strong>All audio clips are provided as uncompressed PCM 16 bit, 16 kHz, mono audio files. <strong>Data split: </strong> The FUSS dataset is partitioned into "train", "validation", and "eval" sets, following the same splits used in FSD data. Specifically, the train and validation sets are sourced from the FSD50K dev set, and we have ensured that clips in train come from different uploaders than the clips in validation. The eval set is sourced from the FSD50K eval split. <strong>Baseline System: </strong>A baseline system for the FUSS dataset is available at dcase2020_fuss_baseline. <strong>License: </strong>All audio clips (i.e., in FUSS_fsd_data.tar.gz) used in the preparation of Free Universal Source Separation (FUSS) dataset are designated Creative Commons (CC0) and were obtained from freesound.org. The source data in FUSS_fsd_data.tar.gz were selected using labels from the FSD50K corpus, which is licensed as Creative Commons Attribution 4.0 International (CC BY 4.0) License. The FUSS dataset as a whole, is a curated, reverberated, mixed, and partitioned preparation, and is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) License. This license is specified in the `LICENSE-DATASET` file downloaded with the `FUSS_license_doc.tar.gz` file. <strong>Notes:</strong> Added in v1.2: FUSS_baseline_dry_model.tar.gz: baseline separation model trained on non-reverberated (dry) data. FUSS_DESED_baseline_dry_2_model.tar.gz:: baseline separation model for the DESED task, trained on a mixture of DESED in-domain data and FUSS data Added in v1.3: FUSS_DESED_baseline_dry_1_model.tar.gz: baseline separation model for the DESED task, trained to separate DESED mixtures from dry FUSS mixtures (DmFm) FUSS_DESED_baseline_dry_4_model.tar.gz: baseline separation model for the DESED task, trained to separate DESED background, dry FUSS mixture, and 5 DESED foreground sources with PIT (PIT) FUSS_DESED_baseline_dry_4np_model.tar.gz: baseline separation model for the DESED task, trained to separate DESED background, 10 DESED classes, and dry FUSS mixture without PIT (Classwise) FUSS_DESED_baseline_dry_6_model.tar.gz: baseline separation model for the DESED task, trained to separate DESED background, 5 DESED foreground sources, 4 dry FUSS sources, with groupwise PIT (GroupPIT) The names in parentheses are the task names from Table 3 of the following paper: Nicolas Turpault, Scott Wisdom, Hakan Erdogan, John R. Hershey, Romain Serizel, Eduardo Fonseca, Prem Seetharaman, and Justin Salamon, "Improving Sound Event Detection in Domestic Environments using Sound Separation", DCASE 2020.

自由通用声音分离（Free Universal Sound Separation, FUSS）数据集是一款用于任意声音分离实验的数据库，包含任意声音混合样本与声源级参考样本。它是DCASE2020挑战赛任务4——家庭环境声音事件检测与分离——的官方声音分离数据集。

<strong>引用说明：</strong>若您使用FUSS数据集或其部分内容，请引用我们发表的关于该数据集与基线系统的论文[1]。由于FUSS基于FSD数据构建，因此同时请引用[2]。

<strong>概述：</strong>FUSS音频数据源自Freesound数据集的预发布版本FSD50K，这是一个基于Freesound平台内容、采用AudioSet本体标注标签的声音事件数据集。借助FSD50K的标签，我们对源音频文件进行了筛选，确保其大概率仅包含单一种类的声音。此类源音频文件不提供标签，且标签不属于挑战赛的评估范围。针对DCASE任务4的声音分离与事件检测挑战赛，参赛系统不得使用FSD50K的标签，即便FSD50K发布后该标签可公开获取。

为生成混合样本，我们将时长10秒的源音频片段与模拟房间冲激响应进行卷积后再叠加。每段10秒的混合样本包含1至4个声源。时长超过10秒的源音频文件被视为“背景”声源，每个混合样本均包含一个全程活跃的背景声源。我们提供了数据集生成脚本、房间冲激响应文件以及原始源音频文件。

<strong>适配DCASE2020挑战赛任务4的设计动机：</strong>本数据集为研究声源分离如何助力声音事件检测（反之亦然）提供了研究平台。过往研究已证明通用声音分离（即任意声音的分离）是可行的[3]，且声音事件检测可辅助通用声音分离任务[4]。但声音分离能否助力声音事件检测，仍有待验证。在嘈杂环境中，声音事件检测难度更高，因此声源分离可作为一项实用的预处理步骤。用于声音事件检测的强标注数据相对稀缺，尤其是在特定领域内限定类别的场景中。与之相对，声源分离数据在训练时无需声音事件标签，且可获取的数量更为充足。在此背景下，本数据集的设计思路是利用规模更大的无标注分离数据训练声源分离系统，将其作为前端模块，辅助基于有限标注数据训练的声音事件检测系统。

<strong>房间模拟：</strong>房间冲激响应采用基于频率相关壁面的镜像法进行模拟。每个冲激响应对应一个随机尺寸、随机壁面材质的矩形房间，其中单麦克风与最多4个声源被随机布置在空间中的不同位置。

<strong>数据生成脚本：</strong>数据生成流程基于scaper工具[5]开发，通过脚本生成带有随机时序的声源事件混合样本，同时包含一段覆盖整个混合片段时长的背景声源。相关脚本已上传至该GitHub仓库。每个混合样本均采用独立的房间模拟流程添加混响，每个声源对应不同空间位置带来的专属混响效果。混响后的混合样本通过叠加各混响声源生成。数据集生成脚本支持自定义修改，以便参赛选手按需重新混合与扩充训练数据。每个混合样本对应的源音频文件也会被单独导出，用作训练与评估的参考样本。

注：我们未对Freesound源数据中的数字静音进行移除处理，因此部分参考声源可能包含数字静音，且存在少量混合样本的背景参考音频完全为数字静音的情况。数字静音在事件识别公开评测数据中同样存在，因此在实际应用中具备处理该情况的能力至关重要。我们的评测脚本会通过忽略所有静音参考声源的方式处理该问题。

<strong>音频格式：</strong>所有音频片段均采用未压缩的PCM 16位、16kHz单声道音频格式。

<strong>数据集划分：</strong>FUSS数据集按照FSD数据的划分规则，被划分为“训练集”“验证集”与“评测集”。具体而言，训练集与验证集源自FSD50K的开发集，且我们已确保训练集与验证集的音频片段来自不同的上传者。评测集源自FSD50K的评测划分子集。

<strong>基线系统：</strong>FUSS数据集的基线系统可通过dcase2020_fuss_baseline获取。

<strong>许可协议：</strong>用于构建FUSS数据集的所有音频片段（即FUSS_fsd_data.tar.gz中包含的内容）采用知识共享CC0协议发布，这些音频均来自freesound.org。FUSS_fsd_data.tar.gz中的源数据通过FSD50K语料库的标签筛选得到，FSD50K采用知识共享署名4.0国际（CC BY 4.0）协议许可。FUSS数据集整体经过整理、添加混响、混合与划分处理，采用知识共享署名4.0国际（CC BY 4.0）协议发布。该协议的详细说明可在`FUSS_license_doc.tar.gz`文件附带的`LICENSE-DATASET`文件中查看。

<strong>补充说明：</strong>
v1.2版本新增内容：
- FUSS_baseline_dry_model.tar.gz：基于非混响（干）数据训练的基线分离模型
- FUSS_DESED_baseline_dry_2_model.tar.gz：针对DESED任务的基线分离模型，基于DESED领域内数据与FUSS数据的混合数据集训练

v1.3版本新增内容：
- FUSS_DESED_baseline_dry_1_model.tar.gz：针对DESED任务的基线分离模型，用于分离DESED混合样本与干FUSS混合样本（DmFm）
- FUSS_DESED_baseline_dry_4_model.tar.gz：针对DESED任务的基线分离模型，用于分离DESED背景、干FUSS混合样本与5个DESED前景声源，采用置换不变训练（Permutation Invariant Training, PIT）
- FUSS_DESED_baseline_dry_4np_model.tar.gz：针对DESED任务的基线分离模型，用于分离DESED背景、10个DESED类别与干FUSS混合样本，不采用置换不变训练（Classwise）
- FUSS_DESED_baseline_dry_6_model.tar.gz：针对DESED任务的基线分离模型，用于分离DESED背景、5个DESED前景声源与4个干FUSS声源，采用分组置换不变训练（GroupPIT）

括号内的名称为下述论文表3中提及的任务名称：Nicolas Turpault, Scott Wisdom, Hakan Erdogan, John R. Hershey, Romain Serizel, Eduardo Fonseca, Prem Seetharaman, and Justin Salamon, "Improving Sound Event Detection in Domestic Environments using Sound Separation", DCASE 2020.