german-english-voice-acting-challenges

# Multilingual Voice Acting Challenges **Synthetic speech with recognized transcripts, timed voice-acting captions, 192 predicted acoustic scores, speaker vectors and MOSS v2 audio tokens.** [Listen to the annotation demo](https://projects.laion.ai/Humanness-Voice-Annotator/) · [Annotator code and format](https://github.com/LAION-AI/Humanness-Voice-Annotator) · [Voice registry](metadata/voices.json) · [Reference-audio coverage](metadata/reference-audio-summary.json) · [Machine-readable release status](metadata/release-status.json) **Migration status: complete. 462,395 complete recordings (3,278.16 hours) have been verified and uploaded in the new layout. The target is 462,395 unique recordings, including 240,000 newly processed English recordings.** Raw generation completion is different from annotation completion. The current status file reports each processing stage separately. Earlier source archives remain available until replacement archives and reference joins pass all checks. ## Collections | Folder | Contents | Complete recordings uploaded in this layout | Final target | |---|---|---:|---:| | [`standard_voices/en_de/`](standard_voices/en_de) | Initial English/German acting catalog | 38,427 | 38,427 | | [`standard_voices/multilingual/`](standard_voices/multilingual) | Five-language acting catalog and physical-situation expansion | 177,808 | 177,808 | | [`standard_voices/emotion_masking_physical/`](standard_voices/emotion_masking_physical) | Balanced English roles, emotion masking, styles and physical situations | 240,000 | 240,000 | | [`character_voices/french_accent/`](character_voices/french_accent) | Playful French-accent character performances in English and German | 4,000 | 4,000 | | [`add_on_voices/multilingual/`](add_on_voices/multilingual) | Separate collection using 24 additional synthetic voices | 2,160 | 2,160 | These are disjoint recording collections. The old token-only export and its corresponding audio are counted once. The additional-voice collection remains separate from the standard 30-voice collections. French-accent performances reuse nine standard identities plus the additional Gaston character identity: this is not 10 extra standard voices. The registry contains **55 distinct identities: 30 standard + 24 add-on + Gaston**. The five-language collection covers English, German, French, Dutch and Spanish. The initial collection covers English and German. The new balanced English collection is English only; do not interpret its English counts as five-language coverage. Current uploaded language counts: de: 45,679, en: 285,679, es: 43,679, fr: 43,679, nl: 43,679. ## The balanced English collection 8,000 performance conditions are rendered once with each of the 30 standard identities (8,000 recordings per voice). They use 7,000 distinct story units: 1,000 masking pairs intentionally share spoken wording. Conditions cover 1,000 roles, 40 emotion/state directions at four intensity levels, 57 VoiceNet dimensions sampled at low/middle/high settings, and physical/social situations. There are 2,000 masking conditions (60,000 recordings): maintained masking and leakage/repair. The 5,130 primary VoiceNet calibration conditions yield 900 recordings per dimension/bin. Other conditions retain emotion/context diversity. Physical context comes from 600 Extreme Physical scenarios, 351 situation tags and 266 compound acting edge cases. These catalogs overlap conceptually and are not interchangeable with mutually exclusive measured event labels. 806 Eric Morris-inspired concepts provide acting direction. A condition can contain multiple intended properties; it is not evidence that the synthesizer realized them. Intended performance, observed predictions and ASR are saved separately. ## What every new training TAR contains | Member | Meaning | |---|---| | `<neutral-id>.mp3` | Original mono audio bytes, preserved without another lossy encoding | | `<neutral-id>.json` | Alias, language, split, hashes, prompt, actual ASR and codec contract | | `<neutral-id>.txt` | The default `GENERAL:` / `SCRIPT:` training prompt | | `<neutral-id>.annotation.json.gz` | Full canonical annotation: global/local predictions, event endpoints, original model outputs and normalization | | `<neutral-id>.codes.npy` | MOSS Audio Tokenizer v2 targets: little-endian int16 `[frames,12]` | `GENERAL:` describes the full performance with the MOSS voice captioner. `SCRIPT:` preserves recognized spoken words and inserts local emotion/style cues with durations, free-form burst captions in parentheses, and pauses in square brackets. JSON retains exact **start and end** times and overlaps. Overlapping bursts do not advance the elapsed script budget twice. Intended generation text is never substituted for recognized speech. ```text GENERAL: A close, warm speaker sounds uncertain at first, then gently amused. SCRIPT: (hesitant; 1.20 seconds duration) I… wasn't sure. [0.30 seconds pause] ("A short, airy chuckle.", 0.40 seconds) (warmer, relieved; 2.00 seconds duration) But I'm glad you came. ``` This example illustrates the format; it is not a human-verified transcript. For executable reading and reference resolution see [code/load_training.py](code/load_training.py). ## Annotation models and versions The open [Humanness Voice Annotator](https://github.com/LAION-AI/Humanness-Voice-Annotator) combines: | Stage | Released model / code | Saved evidence | |---|---|---| | Global and local acoustic predictions | [Humaneness Ears Medium](https://huggingface.co/laion/humaneness-ears-base-medium) | 192 raw scores and reference z-scores; 40 emotions; 57 VoiceNet dimensions; quality, genuineness, blend and extra attributes; 128D timbre and 250D identity predictions; frame/event outputs | | Recognized words and timing | [Parakeet TDT v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) | Actual ASR text, words and sentence alignment | | Whole-recording caption | [MOSS voice-captioner](https://huggingface.co/laion/moss-voice-captioner) | Free-form voice/performance description | | Full-audio interval caption | [MOSS vocal-burst-captioner](https://huggingface.co/laion/moss-vocal-burst-captioner) | Description conditioned on full audio plus start/end timestamps | | Normalization and compact cues | [Procedural voice captions](https://github.com/LAION-AI/procedural-voice-captions) | Frozen reference means/stds, selected local tags and timing rules | | Codec targets | [MOSS Audio Tokenizer v2](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer-v2) | 48 kHz audio, first 12 RVQ codebooks, 80 ms frames, original length and right padding | See [metadata/annotation-models.json](metadata/annotation-models.json) and [metadata/training-codec.json](metadata/training-codec.json) for immutable revisions. The codec uses revision `f6e20e543b33d2c252a7ef71bdf8aa71e5ff9169`. Code values are 0–1023; mono is duplicated into the codec's stereo interface; the final frame is right-padded and that padding must be removed on reconstruction. No codec model is needed to read the saved tokens. Burst defaults are threshold **0.60**, minimum duration **0.08 s**, and overlap suppression at **0.50** of the shorter proposal. Both MOSS adapters decode greedily; global/interval caps are 512/256 tokens with a documented 1,024-token retry. A rare repeated cap triggers a deterministic anti-repetition retry; unresolved records are withheld. Audio inference uses the audio alone, without intended target emotion or intended script context. Captions/scores/vectors are weak machine labels, not human annotations or outputs of the original quality/speaker teachers. In particular, the Orange-style vectors are **Ears predictions**, not freshly computed Orange teacher vectors. DNSMOS and AudioBox fields are likewise student predictions. Model-source attribution is retained. ## Aliases, references and provenance All ordinary training keys, speaker fields and paths use neutral names. The aliases are **synthetic speaker identifiers**, not claims of human identity or human-recorded speech. There are no collisions between the 30 standard aliases, the 24 add-on aliases and Gaston. For every target, [metadata/reference-audio.jsonl.gz](metadata/reference-audio.jsonl.gz) points to a **different recording by the exact same speaker**, with an audio hash, TAR member, corresponding codec member and split. Selection prefers a different dominant **reference-normalized predicted emotion**, a different challenge and the same split/language. Any unavailable emotion contrast or cross-split fallback is explicitly flagged. Strict held-out evaluation should exclude cross-split fallback references. References reuse audio already present in the release; audio is not copied once per target. [The loader](code/load_training.py) resolves and validates them. The [provenance directory](provenance/) retains original generation metadata, identity mappings and source revision information for reproducibility and attribution, separate from ordinary training metadata. This neutral naming convention does not conceal that the corpus is synthetic. Original archives are preserved in the tagged source revision; the latest layout retires them only after complete replacement verification. Audio and existing codec targets are preserved byte-for-byte; changed metadata has newly computed signatures. Matched story/masking variants in the new English collection retain their original train/validation/test lineage split across all voices. Earlier collections retain their historical split assignments. They were created separately and have **not** been certified as jointly scenario-disjoint across collections: use explicit cross-collection deduplication for an independent benchmark. Do not treat generation targets as verified emotion labels. ASR and burst detection can miss or hallucinate content; inspect the review flags and audio. ## License New dataset annotations and release documentation are **CC BY 4.0**, attributed to LAION. Synthetic origin, source provenance and applicable upstream model/code licenses remain documented. This release does not relicense upstream model weights. Attribute the release and preserve its provenance when redistributing it. ## Voice alias catalog These English descriptions paraphrase the frozen generation-time catalogs. Presentation, register, age and accent are catalog characteristics or explicitly marked character directions, **not demographic measurements of real people**. Actual emotion and delivery vary by recording. Unspecified values are left unspecified. | Alias | Group | Voice presentation | Register | Catalog age | Description | |---|---|---|---|---|---| | Aurora | standard_30 | female | high | not specified | Sunny, optimistic presence with a forward-moving delivery. | | Killian | standard_30 | male | low | not specified | Cheerful, energetic and approachable, with a substantial vocal body. | | Ezra | standard_30 | male | low | not specified | Grounded, consistent delivery suited to factual explanation. | | Liora | standard_30 | female | medium | not specified | Decisive and assured, with a direct presentation. | | Jasper | standard_30 | male | low | not specified | Passionate, animated delivery with considerable energy. | | Seraphina | standard_30 | female | high | not specified | Youthful-sounding, lively presence with an energetic delivery. | | Kael | standard_30 | male | low | not specified | Sober and resolute, with a practical, matter-of-fact delivery. | | Esme | standard_30 | female | medium | not specified | Airy, light delivery with an unhurried conversational feel. | | Genevieve | standard_30 | female | medium | not specified | Informal and at ease, with a relaxed conversational manner. | | Freya | standard_30 | female | medium | not specified | Enthusiastic and positive, with an animated promotional presence. | | Gideon | standard_30 | male | low | not specified | Quiet and intimate, with an airy, softly produced tone. | | Jensen | standard_30 | male | low | not specified | Cleanly enunciated, with precise and distinct articulation. | | Soren | standard_30 | male | low | not specified | Unforced, informal delivery with an easy conversational pace. | | Declan | standard_30 | male | low | not specified | Refined and flowing, with a composed professional finish. | | Isolde | standard_30 | female | medium | not specified | Rounded, gentle delivery with a polished professional character. | | Jocelyn | standard_30 | female | medium | not specified | Precisely articulated, intelligible delivery with a clear instructional feel. | | Darius | standard_30 | male | low | not specified | Low, textured delivery with a rugged and weighty dramatic presence. | | Maddox | standard_30 | male | medium | not specified | Measured, explanatory delivery suited to longer narration. | | Persephone | standard_30 | female | high | not specified | Buoyant and energetic, with an optimistic, outgoing presence. | | Elara | standard_30 | female | high | not specified | Delicate, restful delivery with a reassuring, soothing quality. | | Elias | standard_30 | male | low | not specified | Resolute and commanding, with a clear sense of authority. | | Silas | standard_30 | male | low | not specified | Level and dependable, with an even, balanced delivery. | | Kaia | standard_30 | female | medium | not specified | Mature-sounding, assured delivery with a formal professional presence. | | Sloane | standard_30 | neutral | medium | not specified | Assertive and self-possessed, with a direct forward presentation. | | Caspian | standard_30 | male | low | not specified | Welcoming and personable, with an informal, warm conversational tone. | | Thorne | standard_30 | male | low | not specified | Relaxed and informal, with an easy, conversational delivery. | | Celeste | standard_30 | female | medium | not specified | Softly reassuring, with a mild, composed and soothing delivery. | | Rowan | standard_30 | male | low | not specified | Expressive and lively, with an engaging spoken presentation. | | Ryder | standard_30 | male | medium | not specified | Assured and explanatory, with an even, knowledgeable presence. | | Ursula | standard_30 | female | medium | not specified | Reassuring and welcoming, with an inviting, warm delivery. | | Adrian | add_on_24 | male | unspecified | middle aged | Dominant, resolute delivery with a firm presence. | | Alina | add_on_24 | female | unspecified | middle aged | Precise, engaging instructional delivery with a professional finish. | | Bianca | add_on_24 | female | unspecified | middle aged | Luminous, welcoming delivery with a polished professional manner. | | Bastian | add_on_24 | male | unspecified | old | Seasoned-sounding and considered, with balanced, distinct articulation. | | Bennett | add_on_24 | male | low | middle aged | Low, full-bodied resonance with a soothing, reassuring presence. | | Caius | add_on_24 | male | unspecified | middle aged | Textured, husky character delivery with a mischievous edge. | | Calvin | add_on_24 | male | low | young | Low, self-assured delivery with an animated, energetic character. | | Cedric | add_on_24 | male | unspecified | middle aged | Personable and unpretentious, with a relaxed conversational presence. | | Dorian | add_on_24 | male | unspecified | middle aged | Consistent, composed delivery with a formal broadcast character. | | Edmund | add_on_24 | male | low | middle aged | Low, dark-toned dramatic delivery with a serious character. | | Emmett | add_on_24 | male | unspecified | middle aged | Flowing and composed, with a dependable conversational presence. | | Felix | add_on_24 | male | low | old | Low, gravelly texture balanced by a welcoming narrative warmth. | | Gareth | add_on_24 | male | unspecified | middle aged | Enveloping, engaging narrative delivery with a mature-sounding presence. | | Hugo | add_on_24 | male | unspecified | young | Rough-edged, forceful character delivery with youthful-sounding energy. | | Iris | add_on_24 | female | unspecified | young | Lighthearted and radiant, with a welcoming, playful character. | | Leonie | add_on_24 | female | unspecified | young | Animated, individualistic delivery with a lightly cheeky attitude. | | Lucian | add_on_24 | male | unspecified | young | Outgoing, self-assured delivery with a lively presentation style. | | Lydia | add_on_24 | female | unspecified | middle aged | Soft, plush vocal texture with a confident dramatic presence. | | Marielle | add_on_24 | female | unspecified | middle aged | Well-informed, professional delivery with a positive, animated manner. | | Remy | add_on_24 | neutral | unspecified | middle aged | Unhurried and composed, with an understated explanatory delivery. | | Ronan | add_on_24 | male | unspecified | middle aged | Relaxed and informal, with a full, resonant vocal body. | | Sabine | add_on_24 | female | unspecified | young | Assured and reassuring delivery with a polished, mature-sounding presence. | | Tobias | add_on_24 | male | low | middle aged | Low, welcoming narrative delivery with a professional finish. | | Wesley | add_on_24 | male | unspecified | young | Easygoing, positive delivery with an unhurried conversational manner. | | Gaston | character_extra | male | low / warm (character direction) | not specified | Warm, playful and gallant character; outwardly capable, emotionally receptive and lightly comic. Register and manner are performance directions, not acoustic measurements. |