vocalburst-locator-synth-data

# Vocal Burst Locator - Synthetic Training Data Synthetic audio soundscapes with frame-level vocal burst annotations, used to train [laion/vocalburst-locator](https://huggingface.co/laion/vocalburst-locator) — a Whisper-based model that detects and localizes vocal bursts (laughs, coughs, sneezes, sighs, gasps, cries, screams, etc.) in audio. ## Dataset Summary | Split | Samples | Size | |---|---|---| | Train | 32,712 | ~8.4 GB | | Validation | 300 | ~79 MB | | **Total** | **33,012** | **~8.5 GB** | Each sample is a pair of files: - **`sample_XXXXX.mp3`** — a mono audio soundscape (3–30 seconds, 44.1 kHz, 96 kbps MP3) - **`sample_XXXXX.json`** — labels with vocal burst timestamps and metadata ## Label Format ```json { "events": [ {"start_time": 3.21, "end_time": 4.85}, {"start_time": 12.50, "end_time": 13.10} ], "duration_sec": 24.5, "bg_type": "music", "n_vocal_bursts": 2 } ``` | Field | Type | Description | |---|---|---| | `events` | list | Vocal burst segments with `start_time` and `end_time` in seconds | | `duration_sec` | float | Total audio duration in seconds | | `bg_type` | string | Background type: `"music"`, `"sfx"`, `"music+sfx"`, or `"silence"` | | `n_vocal_bursts` | int | Number of vocal bursts in the clip | Samples with `n_vocal_bursts: 0` have an empty `events` list (negative samples). ## Dataset Composition ### Class Balance - **50% negative** (no vocal bursts): 16,506 samples - **50% positive** (1+ vocal bursts): 16,506 samples Distribution of vocal bursts per clip: | VBs per clip | Count | |---|---| | 0 | 16,506 | | 1 | 6,690 | | 2 | 6,063 | | 3 | 2,747 | | 4 | 771 | | 5 | 201 | | 6+ | 34 | Total vocal burst events across the entire dataset: **31,360** ### Background Types | Background | Count | Fraction | |---|---|---| | Music | 10,464 | 31.7% | | SFX (sound effects) | 10,465 | 31.7% | | Music + SFX | 10,433 | 31.6% | | Silence | 1,650 | 5.0% | ### Duration Clip durations follow a beta distribution biased toward longer clips: `Beta(3.0, 1.2) * 27 + 3`, giving a range of 3–30 seconds with most clips near 20–30 seconds. ## How It Was Made The dataset was generated synthetically by mixing vocal burst audio clips on top of background audio. The generation pipeline (`generate_dataset.py` in [laion/vocalburst-locator](https://huggingface.co/laion/vocalburst-locator)) works as follows: ### 1. Source Audio Four source audio collections were downloaded (using `download_sources.py`): | Source | Dataset | Count | Description | |---|---|---|---| | Vocal bursts | [laion/improved_synthetic_vocal_burts](https://huggingface.co/datasets/laion/improved_synthetic_vocal_burts) | 15,680 | Synthetic vocal burst clips (laughs, coughs, sneezes, sighs, gasps, cries, screams, etc.) | | Music | [laion/captioned-ai-music-snippets](https://huggingface.co/datasets/laion/captioned-ai-music-snippets) | 5,000 | AI-generated music snippets | | SFX (AudioSet) | [mitermix/audioset-with-grounded-captions](https://huggingface.co/datasets/mitermix/audioset-with-grounded-captions) | 5,000 | Sound effects from AudioSet (filtered to exclude clips containing vocal bursts or excessive speech) | | SFX (AudioSnippets) | [mitermix/audiosnippets_small_with_detailed_annotation](https://huggingface.co/datasets/mitermix/audiosnippets_small_with_detailed_annotation) | 3,000 | Diverse audio snippets with annotations | ### 2. Soundscape Generation For each sample, the generator: 1. **Picks a duration** from the beta distribution (3–30s) 2. **Selects a background type** (music, SFX, music+SFX, or silence) based on the target distribution 3. **Creates the background** by randomly selecting and mixing source audio at random volumes: - Music: 1 track at 30–100% volume - SFX: 2–8 sound effects at 20–80% volume, placed at random positions - Music+SFX: combination of both - Silence: empty background 4. **For positive samples** (50%), places 1–5+ vocal burst clips at random positions with random volume (10–100%). Some clips may overlap (25% probability). 5. **Applies optional augmentation** (15% of samples): low-pass filtering and/or additive noise 6. **Exports** as mono MP3 (44.1 kHz, 96 kbps) with a JSON label file Each of the 15,680 vocal burst source clips appears exactly **twice** across the entire dataset. ### 3. Train/Val Split - 300 samples are reserved for validation - The remaining 32,712 are used for training - The split is deterministic (seed=2024) ## Using This Dataset ### Loading Samples ```python import json import librosa # Load a single sample audio, sr = librosa.load("train/sample_00042.mp3", sr=16000) with open("train/sample_00042.json") as f: labels = json.load(f) print(f"Duration: {labels['duration_sec']:.1f}s") print(f"Background: {labels['bg_type']}") for ev in labels["events"]: print(f" Vocal burst: {ev['start_time']:.2f}s - {ev['end_time']:.2f}s") ``` ### Converting to Frame Labels The [vocalburst-locator](https://huggingface.co/laion/vocalburst-locator) model operates at 50 fps (1500 frames per 30s). To convert event timestamps to frame-level binary labels: ```python import numpy as np num_frames = 1500 clip_seconds = 30.0 frame_labels = np.zeros(num_frames, dtype=np.float32) for ev in labels["events"]: start_frame = int(ev["start_time"] / clip_seconds * num_frames) end_frame = int(ev["end_time"] / clip_seconds * num_frames) start_frame = max(0, min(start_frame, num_frames - 1)) end_frame = max(0, min(end_frame, num_frames)) frame_labels[start_frame:end_frame] = 1.0 ``` ## Trained Model This dataset was used to train [laion/vocalburst-locator](https://huggingface.co/laion/vocalburst-locator), which achieves: | Metric | Value | |---|---| | Event F1 | 0.752 | | Event Precision | 0.897 | | Event Recall | 0.781 | See the model card for full details on architecture, training, and inference. ## Files Training data is split into shards (subdirectories) of ~4,000 samples each due to hosting platform limits: ``` vocalburst-locator-synth-data/ train/ shard_00/ # samples 00000–03999 (4,000 samples) sample_00000.mp3 sample_00000.json ... shard_01/ # samples 04000–07999 shard_02/ # samples 08000–11999 ... shard_08/ # samples 32000–32711 (712 samples) val/ # 300 samples (flat directory) sample_00000.mp3 sample_00000.json ... manifest.json # Dataset generation metadata ``` ### Loading All Training Data ```python import os, json, glob # Iterate across all shards for shard_dir in sorted(glob.glob("train/shard_*")): for json_path in sorted(glob.glob(os.path.join(shard_dir, "*.json"))): mp3_path = json_path.replace(".json", ".mp3") with open(json_path) as f: labels = json.load(f) # Process mp3_path and labels... ``` ## Generation Parameters | Parameter | Value | |---|---| | Target sample rate | 44,100 Hz | | MP3 bitrate | 96 kbps | | Duration distribution | Beta(3.0, 1.2) * 27 + 3 | | Negative fraction | 50% | | VB volume range | 10–100% | | VB overlap probability | 25% | | Quality augmentation probability | 15% | | Random seed | 2024 | ## License Apache 2.0