Supplementary Material for "Navigable Semantic Sound Maps for Auditory Displays"

This paper presents a method to use t-SNE for training low-dimensional timbre maps obtained from musical recordings in a way that similar sound spectra are represented at similar location within the map. To use such maps for generating novel sounds and timbre trajectories, we use Kernel Regression Mapping for the inverse transformation from map space to timbre space and Griffin-Lim algorithm for phase reconstruction. With this mix of methods, we achieve a way to both visually explore timbral patterns and compress control data. As an application for navigable timbre maps we introduce a novel method - related to and inspired from Wave Space Sonification (WSS) - for the auditory exploration of patterns in multivariate time-series data by using the semantic sound map as the connecting representation. We demonstrate our approach by sonifying ECG data relating to cardiological pathologies.

<strong>Code Repository</strong>
The source code is published as open source in a git repository, available at the GitLab of Bielefeld University here.

<strong>Sound Examples</strong>
Saxophone Synthesis

<strong>S1.1:</strong> <em>Trajectory T1</em> - $\sigma_{krm}$ = 1.0, freq = 220 Hz, duration = 4 seconds
/audio&gt;

<strong>S1.2:</strong> <em>Trajectory T2</em> - $\sigma_{krm}=1.0$, freq = 220 Hz, duration = 2 seconds
/audio&gt;

<strong>S1.3:</strong> <em>Trajectory T3</em> - $\sigma_{krm}=1.0$, freq = 110 Hz, duration = 5 seconds
/audio&gt;

<strong>S1.4:</strong> <em>Trajectory T4</em> - $\sigma_{krm}=1.0$, freq = 220 Hz, duration = 4 seconds
/audio&gt;

<strong>S1.5:</strong> <em>Trajectory T5</em> - $\sigma_{krm}=1.0$, freq = 220 Hz, duration = 1 second
/audio&gt;

<strong>S1.6:</strong> <em>Trajectory T6</em> - $\sigma_{krm}=1.0$, freq = 220 Hz, duration = 3 seconds
/audio&gt;

Cross Synthesis

<strong>S2.1:</strong> <em>Trajectory T1</em> - $\sigma_{krm}$ = 0.8, No resampling (freq = None), duration = 1 second
/audio&gt;

<strong>S2.2:</strong> <em>Trajectory T2</em> - $\sigma_{krm}$ = 0.8, freq = 220 Hz, duration = 2.5 seconds
/audio&gt;

<strong>S2.3:</strong> <em>Trajectory T3</em> - $\sigma_{krm}$ = 0.8, freq = 220 Hz, duration = 2.5 seconds
/audio&gt;

<strong>S2.4:</strong> <em>Trajectory T4</em> - $\sigma_{krm}$ = 0.8, freq = 220 Hz, duration = 2.5 seconds
/audio&gt;

<strong>S2.5:</strong> <em>Trajectory T5</em> - $\sigma_{krm}$ = 0.8, freq = 220 Hz, duration = 2.5 seconds
/audio&gt;

<strong>S2.6:</strong> <em>Trajectory T6</em> - $\sigma_{krm}$ = 0.8, freq = 220 Hz, duration = 2.5 seconds
/audio&gt;

Speech Synthesis

<strong>S3.1:</strong> <em>Trajectory T1</em> - $\sigma_{krm}$ = 1.0, No resampling (freq = None), duration = 2.5 seconds
/audio&gt;

<strong>S3.2:</strong> <em>Trajectory T2</em> - $\sigma_{krm}$ = 1.0, No resampling (freq = None), duration = 1 second
/audio&gt;

<strong>S3.3:</strong> <em>Trajectory T3</em> - $\sigma_{krm}$ = 1.0, No resampling (freq = None), duration = 1 second
/audio&gt;

<strong>S3.4:</strong> <em>Trajectory T4</em> - $\sigma_{krm}$ = 1.0, No resampling (freq = None), duration = 1 second
/audio&gt;

<strong>S3.5:</strong> <em>Trajectory T5</em> - $\sigma_{krm}$ = 1.0, No resampling (freq = None), duration = 1 second
/audio&gt;

<strong>S3.6:</strong> <em>Trajectory T6</em> - $\sigma_{krm}$ = 1.0, No resampling (freq = None), duration = 1 second
/audio&gt;

Signal Reconstruction

<strong>S4.1:</strong> <em>Original preprocessed (zero padding) sample</em>
/audio&gt;

<strong>S4.2:</strong> <em>Phase Reconstruction using <strong>Griffin-Lim</strong></em>
/audio&gt;

<strong>S4.3:</strong> <em>Phase Reconstruction using <strong>Random Phases</strong></em>
/audio&gt;

ECG Sonification

<strong>V5.1:</strong> <em>Anterior weak (fast playback)</em>
/video&gt;

<strong>S5.1:</strong> <em>Anterior weak (fast playback)</em>
/audio&gt;

<strong>V5.2:</strong> <em>Anterior medium (fast playback)</em>
/video&gt;

<strong>S5.2:</strong> <em>Anterior medium (fast playback)</em>
/audio&gt;

<strong>V5.3:</strong> <em>Anterior strong (fast playback)</em>
/video&gt;

<strong>S5.3:</strong> <em>Anterior strong (fast playback)</em>
/audio&gt;

<strong>V5.4:</strong> <em>Inferior weak (fast playback)</em>
/video&gt;

<strong>S5.4:</strong> <em>Inferior weak (fast playback)</em>
/audio&gt;

<strong>V5.5:</strong> <em>Inferior medium (fast playback)</em>
/video&gt;

<strong>S5.5:</strong> <em>Inferior medium (fast playback)</em>
/audio&gt;

<strong>V5.6:</strong> <em>Inferior strong (fast playback)</em>
/video&gt;

<strong>S5.6:</strong> <em>Inferior strong (fast playback)</em>
/audio&gt;

<strong>V5.7:</strong> <em>Anterior weak (slow playback)</em>
/video&gt;

<strong>S5.7:</strong> <em>Anterior weak (slow playback)</em>
/audio&gt;

<strong>V5.8:</strong> <em>Anterior medium (slow playback)</em>
/video&gt;

<strong>S5.8:</strong> <em>Anterior medium (slow playback)</em>
/audio&gt;

<strong>V5.9:</strong> <em>Anterior strong (slow playback)</em>
/video&gt;

<strong>S5.9:</strong> <em>Anterior strong (slow playback)</em>
/audio&gt;

<strong>V5.10:</strong> <em>Inferior weak (slow playback)</em>
/video&gt;

<strong>S5.10:</strong> <em>Inferior weak (slow playback)</em>
/audio&gt;

<strong>V5.11:</strong> <em>Inferior medium (slow playback)</em>
/video&gt;

<strong>S5.11:</strong> <em>Inferior medium (slow playback)</em>
/audio&gt;

<strong>V5.12:</strong> <em>Inferior strong (slow playback)</em>
/video&gt;

<strong>S5.12:</strong> <em>Inferior strong (slow playback)</em>
/audio&gt;