The LibriJump Evaluation Dataset of Intemittent and Moving Speakers

Overview LibriJump is an evaluation dataset for Sound Source Tracking (SST) with a specific focus on intermittent and moving speakers, i.e., speakers that may move when silent. SST is a subtask of spatial scene analysis that estimates the time-varying spatial positions of sound sources in an acoustic scene from a multichannel audio recording. It features three main objectives: Detection: correctly tell when sources are active or inactive. Localization: reliably estimating the spatial positions of sources. Association: maintaining consistent track identities for each source. Regarding intermittent and moving sources, the nature of movement during silence may make the global spatial tracks of the sources either continuous or discontinuous, depending on the predictability of the movement during silence. Unlike existing SST datasets, which mainly feature speakers moving continuously even when silent, LibriJump contains discontinuous speaker tracks, in which speakers may change position unpredictably while silent. This realistic scenario poses significant challenges for SST, in particular regarding the association performance, i.e, the ability of a SST system to maintain coherent track identities over time. The dataset is introduced and described in more detail in: Iatariene, T., Guérin, A., & Serizel, R. (2025). Tracking of Intermittent and Moving Speakers: Dataset and Metrics. Proceedings of the 11th Convention of the European Acoustics Association, Forum Acusticum 2025. Key Characteristics Intermittent speakers: speakers alternate between speech and silence. Discontinuous spatial tracks: speakers may change position during silence (“jumps”). Controlled complexity: speakers are static while active, to simplify localization and detection, and focus on association challenges. First Order Ambisonics (FOA) audio format. Synthetic acoustic scenes with simulated reverberation and background noise. Presentation LibriJump is divided into three subsets depending on the number of speakers present in the scene: LibriJump-1spk: single-speaker scenes. LibriJump-2spk: two-speaker scenes. LibriJump-3spk: three-speaker scenes. Each subset consists of 150 acoustic scenes, each 60 seconds long. Each acoustic scene is described by a multichannel audio recording sampled at 16000 Hz. It consists of a mixture of 1, 2, or 3 speakers (depending on the subset) speaking intermittently and moving only when silent, thus simulating “jumps”. All the LibriJump acoustic scenes are synthetic and rely on external material: LibriSpeech [1] test-clean corpus for speech material. Pyroomacoustics [2] for room acoustics simulation. Freesound [3] for background noise samples. First-Order Ambisonics (FOA) All LibriJump acoustic scenes are in First-Order Ambisonics (FOA) in the ACN-SN3D convention. Given a source signal $s(t)$ emitted from the position $(\theta, \phi)$, where $\theta$ and $\phi$ denote the spherical azimuth and elevation angles, respectively, the FOA representation of this signal encoded from the position $\underline{x}$ is a 4 channel signal $y(\underline{x},t)$ expressed as: $$ y(\underline{x},t)=\left[\begin{array}{c} 1 \ \sin \theta \cdot \cos \phi \ \sin \phi \ \cos \theta \cdot \cos \phi \end{array}\right] s(t) $$ Ground truths For each acoustic scene, the dataset provides the following ground-truth annotations : Ground truth voice activity detection (VAD) of each speaker. Stored in .npy format. Shape (n_speakers, n_time_samples). Silent periods are marked with NaN values. Ground truth tracks of each speaker. Stored in .npy format. Shape (n_time_samples, n_speakers, 2). The tracks are provided in terms of azimuth and elevation angles, referred to as the direction of arrival (DOA). Angle values are in degrees. The azimuth and elevation angles are in the $[-180 , 180]$ and $[-90 , 90]$ ranges, respectively. Silent periods are marked with NaN values. Wet speeches Stored in .wav format. Each speaker’s isolated, reverberated signal corresponds to their contribution in the main recording. Shape (4, n_time_samples) for the 4 FOA channels Ground truth speaker identities. This corresponds to the original LibriSpeech test-clean speaker identity. Relative speaker energy levels Energy of each speaker relative to the first speaker in dB. Folders structure and organization The following folder structure, common to all subsets, is taken. Each subset contains 50 subfolders, each containing the corresponding recording and time-varying ground truth information of 3 acoustic scenes: # note: the "*" denotes either the number 1, 2 or 3 (there is 3 scenes per subfolder). ├─ 1spk/ │ ├─ 1/ │ │ ├─ doas_item_*.npy # ground truth DOAs, shape (n_time_samples, n_speakers, 2) │ │ ├─ vad_item_*.npy # ground truth VADs, shape (n_speakers, n_time_samples) │ │ ├─ item_*.wav # mixture, shape (4, n_time_samples) │ │ └─ wet_0_item_*.npy # ground truth wet speeches, shape (4, n_time_samples) │ ├─ 2/ │ ├─ ... │ ├─ 50/ │└─ data.json ├─ 2spk/ │ ├─ 1/ │ │ ├─ doas_item_*.npy │ │ ├─ vad_item_*.npy │ │ ├─ item_*.wav │ │ ├─ wet_0_item_*.npy │ │ └─ wet_1_item_*.npy │ ├─ 2/ │ ├─ ... │ ├─ 50/ │ └─ data.json ├─3spk/ └─ └─ ... Data.json For each subset, a data.json file in each folder describes all the files, setting one dictionary key per acoustic scene. Each key contains the following elements: Path to the mixture audio recording. Path to the ground truth DOAs. Path to the ground truth VADs. Relative energy levels for each speaker. Speaker information in a subdictionnary (speaker identities and path to the wet speech). Example entry from the 2spk subset: "1": { "path_mixture": "{root}/2spk/12/item_0.wav", "path_vad": "{root}/2spk/12/vad_item_0.npy", "path_doas": "{root}/2spk/12/doas_item_0.npy", "amplitudes": [ -0.0, -5.928267478942871 ], "wet_speech": { "speaker_1": { "id": "4446", "path": "{root}/2spk/12/wet_0_item_0.wav" }, "speaker_2": { "id": "5142", "path": "{root}/2spk/12/wet_1_item_0.wav" } } }, Usage and evaluation framework LibriJump is accompanied by an open-source evaluation framework that implements standard SST metrics, along with processing tools for reading and processing items from the LibriJump dataset. Citing If you use LibriJump in your work, please cite: Iatariene, T., Guérin, A., & Serizel, R. (2025). Tracking of Intermittent and Moving Speakers: Dataset and Metrics. Proceedings of the 11th Convention of the European Acoustics Association, Forum Acusticum 2025. License This dataset is released under the Creative Commons Attribution Non-Commercial International 4.0 (CC BY-NC 4.0) license. References [1] Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015, April). Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 5206-5210). IEEE. [2] Scheibler, R., Bezzam, E., & Dokmanić, I. (2018, April). Pyroomacoustics: A python package for audio room simulation and array processing algorithms. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 351-355). IEEE. [3] Fonseca, E., Pons Puig, J., Favory, X., Font Corbera, F., Bogdanov, D., Ferraro, A., … & Serra, X. (2017). Freesound datasets: a platform for the creation of open audio datasets.