Data and code for "Arousal is envelope, valence is structure: a source-separation study of where musical emotion lives"

Derived data, per-track model predictions, analysis tables, and the analysis pipeline supporting the manuscript Arousal is envelope, valence is structure: a source-separation study of where musical emotion lives (submitted to PeerJ Computer Science). All 3,232 clips of MERGE Audio Balanced were separated into four stems (vocals, drums, bass, other) with Hybrid Demucs hdemucs_mmi, then recombined into 16 conditions to ask where in a mixture the arousal and valence signals actually live. Rather than treating audio as one undifferentiated signal, the study measures both sufficiency (can a stem alone predict emotion?) and necessity (does removing it cost anything?), which dissociate sharply. The sharpest single result comes from the spectral-shuffle null (Control C3), which preserves a recording's energy envelope while destroying all fine spectral structure. On the official split, it reproduces 82% of the mix's arousal R² (0.374 of 0.455) but only 19% of its valence R² (0.033 of 0.179); the artist-disjoint split gives 76% and 15%. Arousal rides largely on the energy envelope while valence requires spectral structure — which is why the intuitive percussion-versus-harmony account of the arousal/valence asymmetry does not survive testing. The condition-level numbers behind this are in table3_artifact_null.csv. Licence Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0). These are adaptations of the MERGE Audio Balanced dataset, which is released under CC BY-NC 4.0; the NonCommercial term is inherited and is not optional. What is not included, and why Audio recordings. Not redistributed. Obtain MERGE Audio Balanced from Zenodo, DOI 10.5281/zenodo.13939205. Separated stems (~52 GB). Derivative works of commercial recordings; not redistributable. Regenerate with separate_stems.py — approximately 5 hours on 12 CPU cores, deterministic (shifts=0, overlap=0.25, no test-time augmentation). Lyric text. Never redistributed, per the source dataset's terms and the copyright in the underlying works. Only derived sentence embeddings are released. The 16 conditions Reference: mix Sufficiency (isolated): vocals, drums, bass, other Necessity (leave-one-out): accompaniment, no_drums, no_bass, no_other Reconstruction controls: remix, residual Spectral-shuffle nulls: shuffle_mix, shuffle_vocals, shuffle_drums, shuffle_bass, shuffle_other Leave-one-out conditions are the sum of the kept stems, never mix minus stem, so that no condition inherits the separator's residual. Every condition received an identical 646-dimensional feature budget: 134 hand-crafted librosa descriptors plus a 512-dimensional CLAP embedding (laion/larger_clap_music). Ridge/logistic regression (primary) and gradient boosting (secondary) predicted arousal, valence, and the four-way Russell quadrant. Results are reported under two splits: MERGE's official 70/15/15 split, and an artist-disjoint split. The latter exists because 70.5% of official-split test tracks share an artist with training data. Pre-registered hypotheses Four hypotheses were stated before any modeling, each with an explicit falsification criterion fixed in advance. They are reported against those criteria regardless of outcome; two of the four did not survive. table2_hypotheses.csv in analysis_tables.zip states each criterion beside its result for both splits. H1 — Arousal is predominantly percussive. Drums and bass predict arousal close to the full mix, and predict arousal better than they predict valence. Falsified if drums+bass predict valence as well as arousal. Outcome: falsified — the drums minus other contrast for arousal is negative, not positive. H2 — Valence requires harmonic or vocal content. Other and vocals dominate valence; drums alone sit near baseline for valence. Falsified if drums alone predicts valence competitively. Outcome: partially supported — other > drums for valence holds on the artist-disjoint split only; the vocal half of the claim is null. H3 — The vocal stem carries the highest emotion information density, i.e. the most predictive power relative to its share of signal energy. Falsified if vocals rank at or below accompaniment after energy normalization. Outcome: survives its stated test (vocals beat accompaniment), but vocals rank last among the four individual stems on energy-normalized density. So the test it was given was weaker than the claim it was meant to check. H4 — Vocal acoustics and lyric semantics are complementary, not redundant. Combining the two channels beats either alone. Falsified if the combination fails to beat the better single channel. Outcome: not supported — the combination wins in 1 of 8 cells, and the two channels' prediction errors correlate at r = 0.69–0.88. Controls C1 — Loudness confound. Separated stems differ greatly in level, and level alone correlates with arousal. Every condition is LUFS-normalized before feature extraction, and the entire analysis is repeated at natural level. Both passes are deposited; every hypothesis verdict is unchanged between them. C2 — Reconstruction check. Demucs does not reconstruct its input exactly (~26–34 dB reconstruction SNR). The remix condition tests whether the stem sum is equivalent to the mix; the residual condition tests whether the discarded leftover carries emotion. Both are deposited as ordinary conditions, not assumed away. C3 — Separation-artifact null. A stem could appear predictive because the separator imprints label-correlated artifacts. The spectral-shuffle null preserves each condition's energy envelope while destroying pitch, harmony, and timbre; predictive power surviving it is trivial energy information, not musical content. C4 — Silent or absent stems. Instrumental tracks yield a near-silent vocal stem, and quiet passages a near-silent bass. A stem counts as near-absent when its RMS energy falls below 1% of the mix's (energy_share < 0.01). This is stricter than the row-level silent flag, which marks a condition measuring below −70 LUFS that was left unamplified during normalization; a stem can be audible enough to escape silent and still be near-absent. Prevalence is reported per quadrant, and the key analyses re-run excluding affected tracks — otherwise "vocals predict emotion" could partly mean "presence of vocals predicts emotion", a much weaker claim. The threshold is applied in phase8_tables.py; results are in table4_absent_stems.csv and analysis_silent_sensitivity.csv. C5 — Feature parity. Every condition receives an identical 646-dimensional feature budget from an identical extractor, so no condition can win by having more features. Files stem_features.parquet (94.6 MB) — primary analysis input. 35,552 rows = 3,232 songs × 11 conditions, all loudness-normalized to −23 LUFS (EBU R 128 / ITU-R BS.1770). Columns: song_id (string) — MERGE track identifier; joins to every other file condition (string) — one of the 11 conditions above handcrafted (list<float>) — 134-d librosa descriptor vector clap (list<float>) — 512-d CLAP embedding lufs (float) — integrated loudness before normalization gain_db (float) — gain applied to reach −23 LUFS, capped at +40 dB silent (bool) — condition measured below −70 LUFS; left unamplified rms_raw (float) — RMS at natural level energy_share (float) — this condition's RMS energy as a fraction of the mix error (string) — non-null if extraction failed for this row stem_features_raw.parquet (94.2 MB) — identical schema and row count, extracted at natural stem level with no loudness normalization. This is Control C1. stem_features_shuffle.parquet (43.7 MB) — identical schema. 16,160 rows = 3,232 songs × 5 spectral-shuffle conditions. This is Control C3. The shuffle permutes each frame's magnitude STFT bins independently and randomizes all phases, preserving the per-frame energy envelope at ~12 ms resolution while destroying pitch, harmony, and timbre. Seeds are deterministic per song. lyrics_features.parquet (7.0 MB) — 2,075 rows, one per track with lyrics. Columns: song_id (string), lyrics_sbert (list<float>, 768-d all-mpnet-base-v2 sentence embedding), lyrics_word_count (int). No lyric text; embeddings only. predictions.zip (2.7 MB) — per-track test-set predictions, seven parquet files sharing the schema split, featset, model, condition, target, song_id, y_true, y_pred: test_predictions_reg.parquet — 127,644 rows, regression, normalized conditions test_predictions_reg_raw.parquet — 127,644 rows, regression, natural level (C1) test_predictions_reg_shuffle.parquet — 58,020 rows, regression, shuffle nulls (C3) test_predictions_clf.parquet — 63,822 rows, quadrant classification test_predictions_clf_raw.parquet — 63,822 rows, classification, natural level test_predictions_clf_shuffle.parquet — 29,010 rows, classification, shuffle nulls rq4_test_predictions.parquet — 8,568 rows, RQ4 subset; channel is audio, text, or both These are sufficient to reproduce every bootstrap interval and paired comparison in the paper without refitting a single model. analysis_tables.zip (0.2 MB) — 25 CSVs: paired-bootstrap comparisons against mix and between condition pairs, condition rankings, control results, energy-normalized density (H3), silent-stem sensitivity (C4), artifact-null comparisons (C3), headline metrics, the RQ4 tables, and manuscript Tables 1–5. Files ending _raw are the natural-level counterparts. metadata.zip (0.5 MB) — manifest.parquet, 3,232 rows: track IDs, continuous arousal and valence, quadrant label, artist, title, year, duration, both split assignments, and has_lyrics. Paths are repository-relative. Plus separation_log.parquet, 3,232 rows: per-track sample rate, duration, per-stem RMS, reconstruction error, and wall-clock seconds. code.zip — the analysis pipeline end to end: manifest construction, stem separation, condition synthesis, feature extraction, model fitting, the paired-bootstrap analysis, and the scripts that derive Tables 1–5 and Figures 1–6. Every number and figure reported in the paper can be regenerated from the deposited data with these scripts alone. Omitted are only the scripts that typeset the manuscript and assemble this archive, neither of which produces a reported result. MANIFEST.csv — an inventory with byte sizes and SHA-256 checksums for every file. Reproducing the analysis Starting from this deposit, without re-separating audio, run in order: phase4_models.py (fit models), phase5_analysis.py (paired bootstrap), phase6_figures.py (Figures 1–6 and Table 1), phase7_rq4.py (the voice-versus-words sub-study), phase8_tables.py (Tables 2–5). The order matters: phase 6 consumes phase 5's output, and phase 8 consumes both phase 5 and phase 7. Together, these regenerate every number and figure in the paper. The finished tables are also deposited directly in analysis_tables.zip, so results can be inspected without rerunning anything. To rebuild from audio, obtain MERGE Audio Balanced and run build_manifest.py, separate_stems.py, and extract_features.py first. Scripts contain absolute paths that will need adjusting for your machine. Key dependencies: torch, demucs, librosa, transformers, sentence-transformers, pyloudnorm, scikit-learn, pandas, pyarrow, matplotlib. Citation This deposit, all versions: DOI 10.5281/zenodo.21866703. This specific version: DOI 10.5281/zenodo.21866704. The manuscript cites the all-versions DOI. Cite the manuscript, and cite the MERGE dataset: Louro PL, Redinho H, Santos R, Malheiro R, Panda R, Paiva RP. 2026. MERGE: a bimodal audio-lyrics dataset for static music emotion recognition. IEEE Transactions on Affective Computing 17(2):2058–2075. DOI 10.1109/TAFFC.2026.3668037.