dramabox-reinterpretations-top3

# DramaBox reinterpretations — reward-ranked top 3 of 64 **~10,000 acting prompts, each re-performed 64 times by [`laion/moss-tts-local-transformer-4.55b-voice-acting-v2`](https://huggingface.co/laion/moss-tts-local-transformer-4.55b-voice-acting-v2), with the three highest-reward takes published here.** Roughly 30,000 clips. The companion release [`…-raw64`](#) keeps all 64 candidates per group, so the full reward distribution — not just its upper tail — stays available. ## ⚠️ Read this first: the texts are partly gibberish The scripts come from a synthetic acting corpus and are **a mix of English and German with many disfluencies, mid-word breaks and non-sequiturs**. A typical line: > *"Morgan, we are in a negotiation of physical gravity and climbing, which, while just div- > any negotiation…"* This is a property of the source material, faithfully reproduced by the reinterpretation. Practical consequences: - **Good for**: ASR training (the transcripts are accurate against difficult audio), prosody and expressive-delivery modelling, vocal-burst detection, best-of-N reward research. - **Bad for**: anything that assumes the text is well-formed language. Median WER against the intended script is **0.17**, and only **1.6 %** of candidates transcribe exactly. That is largely the disfluent source text, not model failure — the acoustic quality is much better than the WER suggests. ## Scope — please read the number carefully This release is the **edge-case half**: **10,000 groups / 30,009 clips**, spread over **262 distinct edge-case categories** (`ec000` … `ec261`), roughly evenly. The **second half** — 9,980 groups over the non-edge DramaBox material (`prod1m`, the Gemma-authored acting prompts, stratified by `pathway`) — is generating now and will be added to this same repository, bringing the corpus to **~20,000 groups**. Until that lands, cite the number you actually see in `manifest.parquet` rather than 20k. ## Contents | path | what | |---|---| | `data/top3-NNNN.tar` | WebDataset shards: one `.mp3` + one `.json` per clip | | `data/top3-NNNN.parquet` | per-shard manifest (key, reward, WER, duration, blend, genuineness, burst count) | | `manifest.parquet` | all shards concatenated | Key format: `<src_id>.top<rank>`, rank ∈ {1,2,3} by descending reward. ### Per-clip JSON `reward_v1c`, `reward_parts` (`s`/`t`/`b`/`g`), `rank_in_group` (position among all 64), `wer`, `duration`, `asr_text`, `words` (word-level timestamps), `caption`, `blend_0_10`, `genuineness_0_6`, `emo_sim`, `emonet` (40 dims), `voicenet` (57 dims with regression value, bucket and natural-language label), `quality` (4 dims), `bursts` (spans with start/end/duration and type), plus the full prompt (`prompt_caption`, `prompt_general`, `prompt_script`) and the source identity (`src_id`, `src_duration`, `category`). **Source audio is not duplicated here** — `src_id` resolves against the DramaBox staging corpus. ## The LoRAs used The reinterpretations were generated with the **v3 emotion LoRA adapters**, rank 32 / alpha 64, trained against the same v2 base: **[`TTS-AGI/moss-emotion-loras-v3`](https://huggingface.co/TTS-AGI/moss-emotion-loras-v3)** — 40 emotions, one adapter directory each. Eight of them (`Anger`, `Fatigue_Exhaustion`, `Fear`, `Malevolence_Malice`, `Pain`, `Sadness`, `Sexual_Lust`, `Teasing`) drive the edge-case half of this corpus, hot-swapped per group; adapter swap costs ~0.021 s over 268 modules, so switching between groups is free relative to generation. Base model: **[`laion/moss-tts-local-transformer-4.55b-voice-acting-v2`](https://huggingface.co/laion/moss-tts-local-transformer-4.55b-voice-acting-v2)**. These adapters are trained against **v2** and will not behave correctly on the earlier checkpoint. 🎧 **Listen:** [25 originals vs their top-3 reinterpretations](https://projects.laion.ai/laion-moss-local-1.5-voice-acting-4.55b/reinterpretations_25.html) ## The reward ``` R = (1.00·s + 1.25·t + 1.00·b + 1.00·g) / 4.25 · (1 − min(WER, 1)) s = (cos + 1)/2 over a 42-dim profile (40 EmoNet dims + arousal + valence), measured against the SOURCE clip — "does it re-act the same emotion" t = sigmoid(target emotion, z-scored WITHIN the 64-candidate group) b = sigmoid(vocal-burst blend, corpus robust-z) g = sigmoid(genuineness, corpus robust-z) ``` Two design choices worth knowing, because both were arrived at by measurement: **1. The WER gate is multiplicative, never `/(1 + WER)`.** About **three quarters of candidates have a negative core score**, and dividing a negative number by a larger denominator makes it *larger* — so the division form literally rewards transcription errors on most of the pool. This was confirmed independently on a separate 1,000-clip set, where a literal `WER × quality` filter put **602 of 1,000 candidates at exactly score 0** (60 % had WER 0.00) and therefore selected the half with *worse* transcription. Squashing each component through a sigmoid first makes the core strictly positive, so `× (1 − WER)` is monotone decreasing in WER for every candidate. **2. The target emotion is z-scored *within* each group, not against the corpus.** 20 of the 40 EmoNet dimensions have a corpus MAD ≤ 0.02 and five are exactly 0.0 — the corpus is overwhelmingly *not* any given emotion, so a corpus robust-z divides by ≈0 and explodes. Within group is also the right semantics: the task is ranking inside a group. Blend and genuineness are well-conditioned (MAD 0.625 and 1.601 over 256k clips) and keep corpus robust-z so they stay comparable across groups. ## Observed distributions Over a 38,400-candidate sample of the full pool (i.e. before top-3 selection): | | mean | p10 | median | p90 | |---|--:|--:|--:|--:| | `reward_v1c` | 0.411 | 0.265 | 0.424 | 0.560 | | WER | 0.329 | 0.067 | 0.170 | 0.444 | | `blend_0_10` | 2.154 | 0.000 | 1.656 | 4.966 | | `genuineness_0_6` | 1.656 | 0.635 | 1.503 | 2.945 | | duration (s) | 15.45 | 9.44 | 14.40 | 21.36 | **46.8 %** of candidates contain at least one detected vocal burst (mean 0.81 per clip). ## Annotation - **ASR: CrisperWhisper 2 large**, verbatim mode, word-level timestamps. Chosen over faster models specifically because the transcripts are a deliverable here, not just a ranking signal. - **Vocal bursts**: located first, then classified — with a **200 ms** minimum duration floor. **The span timestamps are the reliable part; the class names are a weak model prior.** The classifier names an already-isolated burst and is known to be unreliable on bursts embedded in continuous speech, where it drifts toward sigh/gasp/ahem. Filter on span timing, treat the label as a hint. - **Emotion/voice attributes**: MLP heads over a VoiceCLAP encoder (EmoNet 40, VoiceNet 57, quality 4, genuineness, blend). Model outputs, not human ratings. ## Caveats - Everything here is **synthetic audio** generated by a TTS model, including any apparent background or recording character. - Scores are model outputs throughout; no human evaluation was performed on this corpus. - Top-3 selection means this release is the **upper tail** of the reward distribution by construction. For unbiased analysis use the raw-64 companion. - The `top3` flag was verified against a fresh ranking by `reward_v1c` on 400 groups — **400/400 agreement**. ## License `other`. Derived from synthetic generations over the DramaBox acting-prompt corpus. Check your own jurisdiction before use beyond research.