character-voices

# Character Voices — DramaBox-annotated Echo-TTS (WER = 0) **What this is.** A collection of **2,184 short character-voice speech clips** spanning **14 distinct character archetypes** (dragon, goblin, gnome, orc, fairy, zombie, robot, cartoon characters, …) × the **40 emotions of the [EmoNet](https://huggingface.co/datasets/laion/EmoNet) taxonomy**. Every clip is paired with a rich JSON record whose centrepiece is a **DramaBox performance prompt** — a vivid, second-person acting brief written by **Gemini** that embeds the exact spoken line. The dataset is intended for training and evaluating **emotion- and character-controllable text-to-speech** systems. **How it was made (one line per stage).** Each archetype is defined by a single short reference voice clip. From that clip: 1. **Write lines** — Gemini listens to the reference voice and writes 3 emotion-coloured utterances for each of the 40 emotions (120 lines/character). 2. **Synthesize** — [Echo-TTS](https://github.com/LAION-AI/scaled-echo-tts) generates 5 duration variants per line, conditioned on the emotion via an AdaLN-0 vector. 3. **Score & filter** — Parakeet-TDT-0.6B-v3 ASR + silence-aware trimming + a speech-quality model rank the candidates; **only clips with word-error-rate = 0.00** survive. 4. **Voice-convert** — Chatterbox converts the winner to the character's reference timbre. 5. **Enhance** — RE-USE (SEMamba) denoise → LavaSR bandwidth-extension to 48 kHz → 160 kbps mono MP3. 6. **Annotate** — Gemini listens to the final audio and writes the **DramaBox prompt** + perceived emotions + archetype descriptor. 7. **Package** — each character becomes a folder *and* a WebDataset tar. > The **complete, runnable pipeline code** that produced this dataset lives in > [`code/`](./code) — including a standalone inference script, two example reference > voices (zombie + orc), and a step-by-step reproduction guide in > [`code/README.md`](./code/README.md). --- ## Repository layout ``` README.md ← you are here <character>/ ← loose files: <key>.mp3 + <key>.json (one pair per sample) wds/<character>.tar ← the SAME pairs packed as a WebDataset shard code/ ← all pipeline + inference code (see code/README.md) ``` Both layouts contain identical data — use the **loose folders** for browsing/random access, or the **`wds/` tars** for high-throughput streaming with `webdataset`. In a tar, each sample's `.json` and `.mp3` share a key and are stored adjacently. ## The per-sample metadata (`<key>.json`) Every audio clip `<key>.mp3` (160 kbps mono) has a sibling `<key>.json` with these fields: | field | type | meaning | |-------|------|---------| | `key` | string | unique sample id, e.g. `b1_01_Amusement_1` (`<batch>_<id>_<emotion>_<variant>`) | | `archetype` | string | character/archetype name, e.g. `charming flirty woman`, `robot` | | `archetype_descriptor` | string | short free-text description of the character/voice | | `emotion` | string | EmoNet emotion key (underscore form), e.g. `Malevolence_Malice` | | `emotion_display` | string | human-readable emotion name | | `transcription` | string | the **intended** line that was synthesized (ground-truth text) | | `asr` | string | Parakeet ASR of the final audio (equals `transcription` since WER = 0) | | **`dramabox_prompt`** | **string** | **the Gemini performance brief (see below)** | | `perceived_emotions` | list[str] | emotions Gemini actually *heard* in the clip (3–5 tags) | | `audio` | string | the mp3 filename for this record | | `model` | string | annotator model id (`gemini-3-flash`) | ### Where the DramaBox prompt is, and its format The DramaBox prompt is the **`dramabox_prompt`** string field, present in **every** `<key>.json` (both in the loose `<character>/` folders and inside `wds/<character>.tar`). **Format / contract.** It is a single plain-text string combining, in *roughly equal measure*: - **(a) performance direction** in vivid **second person** — *who* the character is (the archetype), the **emotion(s) audibly present**, the voice/talking style, and *how* to deliver the line; and - **(b) the exact spoken line**, word-for-word, written as direct speech inside **double quotes** (`"..."`). The quoted line equals `transcription`. **Example** (from `charming flity woman/b1_01_Amusement_1.json`): ```json { "key": "b1_01_Amusement_1", "archetype": "charming flity woman", "emotion": "Amusement", "transcription": "Oh, you really think you can challenge me to a game like that? You are absolutely hilarious when you try to be so serious.", "asr": "Oh, you really think you can challenge me to a game like that? You are absolutely hilarious when you try to be so serious.", "dramabox_prompt": "You are a charming flirty woman. In a light, teasing voice filled with genuine amusement, lean in with a confident smirk and say: \"Oh, you really think you can challenge me to a game like that? You are absolutely hilarious when you try to be so serious.\"", "perceived_emotions": ["Playful", "Amused", "Confident", "Flirtatious"], "archetype_descriptor": "charming flirty woman", "audio": "b1_01_Amusement_1.mp3", "model": "gemini-3-flash" } ``` So a typical TTS training pair is **`dramabox_prompt` (input) → `<key>.mp3` (target)**, or, for plain text-to-speech, **`transcription` → audio**. ## Characters | character (`archetype`) | folder / `wds/*.tar` | samples | |---|---|---| | charming flity woman | `charming_flity_woman` | 200 | | clichee-goblin | `clichee-goblin` | 166 | | cuddle-gnome | `cuddle-gnome` | 176 | | Fairy-2 | `Fairy-2` | 153 | | Fluffy Cookie Monster | `Fluffy_Cookie_Monster` | 125 | | Goblin - en | `Goblin_-_en` | 162 | | monsterous-orc | `monsterous-orc` | 169 | | Old noble Dragon | `Old_noble_Dragon` | 181 | | cute cartoon animal | `cute_cartoon_animal` | 172 | | cartoon gnome | `cartoon_gnome` | 151 | | Zombie-Chris2 | `Zombie-Chris2` | 136 | | Zombie-Chris3 | `Zombie-Chris3` | 147 | | zombie-ref | `zombie-ref` | 145 | | robot | `robot` | 101 | **Total: 2,184 samples across 14 characters**, each labelled with one of 40 EmoNet emotions.