more-synthetic-vocalbursts-raw

# More Synthetic Vocal Bursts (Raw) Synthetic vocal burst audio samples generated from a taxonomy of 202 vocal burst types across multiple text-to-audio and TTS models. Each sample is a short (3–10 second) non-speech vocalization — laughs, cries, gasps, sighs, growls, etc. — generated from text prompts describing the burst type, gender, and age group. ## Models Used | Model | Type | Samples | Sample Rate | Notes | |-------|------|---------|-------------|-------| | **DramaBox** (ResembleAI/Dramabox) | TTS DiT | 2000 | 44.1 kHz | Original DramaBox DiT v1, cfg=2.5, stg=1.5, 30 steps | | **Stable Audio 3 Small SFX** (cocktailpeanut/stable-audio-3-small-sfx) | Text-to-Audio | 2000 | 44.1 kHz | SFX model, 8 steps, cfg_scale=1.0 | | **MOSS SoundEffect v2** (OpenMOSS-Team/MOSS-SoundEffect-v2.0) | Text-to-Audio DiT 1.3B | 781 | 48 kHz | 100 steps, cfg_scale=4.0, bfloat16 | All outputs also have **RE-USE enhanced** versions processed through [NVIDIA RE-USE](https://huggingface.co/nvidia/RE-USE) (9.6M param speech enhancement model). ## Dataset Structure All files are in [WebDataset](https://github.com/webdataset/webdataset) `.tar` format. Each shard contains pairs of `.wav` (audio) and `.json` (metadata) files. ``` dramabox/ # DramaBox TTS outputs (2000 samples, 4 shards) shard-0000.tar shard-0001.tar shard-0002.tar shard-0003.tar sa3/ # Stable Audio 3 Small SFX outputs (2000 samples, 4 shards) shard-0000.tar ... moss/ # MOSS SoundEffect v2.0 outputs (781 samples, 2 shards) shard-0000.tar shard-0001.tar dramabox-reuse/ # DramaBox + RE-USE enhanced (2000 samples, 4 shards) sa3-reuse/ # SA3 + RE-USE enhanced (2000 samples, 4 shards) moss-reuse/ # MOSS + RE-USE enhanced (781 samples, 2 shards) nsfw/ # NSFW comparison samples (78 per model variant) nsfw-original.tar # DramaBox original nsfw-sulfur.tar # DramaBox Sulfur DiT variant nsfw-sa3.tar # SA3 nsfw-moss.tar # MOSS nsfw-original_reuse.tar # + RE-USE enhanced versions nsfw-sulfur_reuse.tar nsfw-sa3_reuse.tar nsfw-moss_reuse.tar ``` ## Metadata Format Each `.json` sidecar contains: ```json { "id": 42, "prompt": "A teenage girl performing belly_laugh, A deep, uncontrollable laugh...", "duration_s": 5.2, "gender": "female", "age_group": "teenage_girl", "vocal_burst_key": "belly_laugh", "vocal_burst_description": "A deep, uncontrollable laugh that involves the whole body..." } ``` ## Taxonomy The vocal burst taxonomy covers 202 entries organized by category: - **Laughter** (8 types): belly laugh, chuckle, giggle, cackle, snicker, etc. - **Crying & Distress** (10 types): sobbing, whimpering, wailing, etc. - **Breathing & Sighs** (11 types): heavy panting, exasperated sigh, etc. - **Surprise & Shock** (6 types): startled yelp, dramatic gasp, etc. - **Disgust & Disapproval** (8 types): retching, scoff, tsk, etc. - **Pain & Discomfort** (8 types): sharp yelp, prolonged groan, etc. - **Effort & Exertion** (8 types): heavy lifting grunt, battle cry, etc. - **Communication Signals** (11 types): shush, psst, wolf whistle, etc. - **Eating & Drinking** (6 types): slurping, lip smacking, etc. - **Sleep & Unconscious** (5 types): snoring, sleep talking, etc. - **Animal Imitations** (6 types): growling, purring, hissing, etc. - **Musical & Rhythmic** (7 types): humming, beatboxing, etc. - **Nervous & Anxious** (7 types): nervous laughter, teeth chattering, etc. - **Age-Specific** (6 types): baby cooing, elderly wheeze, etc. - **Bodily Functions** (9 types): hiccup, sneeze, burp, etc. - **Whistling** (6 types): casual whistle, wolf whistle, etc. - **Oral/Mouth Sounds** (8 types): tongue click, teeth sucking, etc. - **Throat Sounds** (7 types): throat clearing, gargling, etc. - **Nasal Sounds** (5 types): sniffling, snorting, etc. - **Vocal Tics & Reflexes** (6 types): hiccup, involuntary yelp, etc. - **Temperature & Environment** (4 types): shivering chatter, heat exhaustion panting, etc. - **Expressive Interjections** (7 types): eureka exclamation, frustrated argh, etc. - **NSFW** (22 types): intimate vocalizations for adult content The SFW version (180 entries, NSFW removed) is available in the [Voice-Acting-Pipeline repo](https://github.com/LAION-AI/Voice-Acting-Pipeline/blob/main/data/vocal_bursts_taxonomy_sfw.json). ## Demographics Samples span 10 age/gender groups: - Female: toddler girl, pre-puberty girl, teenage girl, young adult woman, middle-aged woman - Male: toddler boy, pre-puberty boy, teenage boy, young adult man, middle-aged man ## Usage ```python import webdataset as wds import soundfile as sf import io dataset = wds.WebDataset("dramabox/shard-{0000..0003}.tar") for sample in dataset: audio_bytes = sample["wav"] metadata = json.loads(sample["json"]) audio, sr = sf.read(io.BytesIO(audio_bytes)) print(f"{metadata['vocal_burst_key']} - {metadata['gender']} - {sr}Hz - {len(audio)/sr:.1f}s") ``` ## Related Resources - [Taxonomy JSON (full)](https://github.com/LAION-AI/Voice-Acting-Pipeline/blob/main/data/vocal_bursts_taxonomy.json) - [Taxonomy JSON (SFW)](https://github.com/LAION-AI/Voice-Acting-Pipeline/blob/main/data/vocal_bursts_taxonomy_sfw.json) - [NVIDIA RE-USE](https://huggingface.co/nvidia/RE-USE) — speech enhancement model used for postprocessing - [MOSS SoundEffect v2.0](https://huggingface.co/OpenMOSS-Team/MOSS-SoundEffect-v2.0) - [Stable Audio 3 Small SFX](https://huggingface.co/cocktailpeanut/stable-audio-3-small-sfx) ## License CC-BY-4.0