Supplementary Material for "pya AGen - Audio Generators for Sonification in Python"

img {
max-width: 100%;
height: auto;
}
video {
max-width: 100%;
height: auto;
}
audio {
width: 100%;
}
/* .scrollx {
overflow-x: scroll;
} */

# PyA - AGen - Sonification Examples

**Supplementary Material for L. Born & T. Hermann (2025). "pya AGen - Audio Generator Extensions for Sonification in Python", Proc. ICAD 2025**
<br>

To install pya AGen, install the feature-agen branch of the pya repository using this command:
```
pip install git+https://github.com/interactive-sonification/pya.git@feature-agen[agen]
```

- pya_agen is an extension for [pya](https://pypi.org/project/pya/), the python audio coding package.
- pya_agen provides flexible ways to generate audio signals using AGens, i.e. audio generators.
- AGens 'lazily' generate (i.e. compute when needed) audio samples on demand.
- For documentation of AGens check out the [example notebook](https://github.com/interactive-sonification/pya/blob/feature-agen/examples/pya-examples-agen.ipynb).
- This Jupyter notebook contains supplementary material for the paper.
- The examples demonstrate the use of pya and Agen in particular for coding sonifications.

Load Datasets for sonification examples

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code>data_path = "https://raw.githubusercontent.com/interactive-sonification/sonecules/main/notebooks/data"
# load eeg dataeeg_df = pd.read_csv( f"{data_path}/epileptic-eeg.csv", delimiter=",", header=None)
# load ecg dataecg_df = pd.read_csv( f"{data_path}/ecg-200Hz-10s-6channels.csv", delimiter=" ", header=None)
# load the penguin data setpenguins_df = sns.load_dataset("penguins")penguins_df = penguins_df.dropna(subset=["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g", "sex"]).reset_index(drop=True)
# load the building data setbuilding_df = pd.read_csv(f"{data_path}/building.csv", delimiter=",", names=[ "monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday", "hour_from_noon", "hour", "am_pm", "temperature", "humidity", "solar_radiation", "wind_speed", "hc_wb_electrical", "hc_wb_cold_water", "hc_wb_hot_water"])</code></pre>

## 1. Audification with pya Agen
Audification is so straightforward that it already works with pya (without Agen): as such

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code># load data into Asiga1 = Asig(eeg_df[5], sr=250)
# plot and audify (aka: play) data at given compression ratea1.plot(lw=1).norm().stereo().fade_out(0.01).play(rate=20) </code></pre>

Your browser does not support the audio element.

<br>

Using AGens we can achieve it using the PlayAsig AGen:

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code># load data into Asigdasig = Asig(eeg_df[[3,5]], sr=250).resample(target_sr=44100)
# define Agen to play data at constant rate asyn = PlayAsig(dasig, rate=20)
# render sound signal (aka play asyn), and plot and play asig = asyn.gen_asig().plot(lw=0.5, offset=2).play(onset=0.5)</code></pre>

Your browser does not support the audio element.

As reward we can now easily solve more complicated audification needs.
For example
- audification of EEG data, but only time segment [3sec, 40sec] with channels 3 left, channel 5 right
- audify the data 'looped'
- the compression rate should jump (at 5 Hz) between 30 and 60 so that we can hear patterns in the data at two rates in a multiplexed way
- render, plot and play 3.4 seconds for the user, with a 0.1s fade in/out to avoid pops and clicks

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code># prepare data data = Asig(eeg_df[[3,5]], sr=250)[{3:40}].resample(target_sr=44100)
# define AGen involving a modulator for raterate_agen = LFPulse(freq=5, width=0.5).linlin(-1,1,30,60)asyn = PlayAsig(data, rate=rate_agen, done="loop").fade_in(0.1).fade_out(0.1)
# render sound signal (aka play asyn) and plot and playasyn.gen_asig(seconds=3.4).plot(lw=0.5, offset=2).play(onset=0.5)</code></pre>


Your browser does not support the audio element.

## 2. Parameter Mapping Sonification with pya AGen
### 2.1. Discrete Parameter Mapping Sonification
For this demo we use the penguins dataframe containing attributes bill length bill depth flipper length and body mass for 330 penguins

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code>df = penguins_dffig, axes = plt.subplots(1, 3)for i, col in enumerate(['bill_length_mm','bill_depth_mm','flipper_length_mm']): df.plot(x=col, y="body_mass_g", kind="scatter", s=2, ax=axes[i])</code></pre>


As minimal example let's play the middle plot with bill_depth as onset and body_mass_g as frequency.
- Note that such as PMSon, including synth definition, mapping and rendering is a two-liner.

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code>SeqAGen([ (pam.linlin(x, 13, 23, 0, 5), (SinOsc.ar(freq=y*0.1) * 0.05 * Line(1,0,0.3))|Pan2.p(pos=0)) for x, y in df[['bill_depth_mm', 'body_mass_g']].to_numpy()]).gen_asig().play()</code></pre>

Your browser does not support the audio element.

<br>

For an example of a more generic and complete discrete parameter mapping sonification, let's
- map body mass to onset
- map bill length to pitch
- map bill depth to stereo position
- map flipper length to duration

For that we define a function asyn that returns a custom Agen.
We can regard it as an synth.

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code>def asyn(pitch=40, amp=0.2, att=0.001, dur=0.2, pos=0): sig = SinOsc.ar(pam.midi_to_cps(pitch)) env = Env([0, 1, 0], [att, dur], done="stop") agen = Pan2(sig*env*amp, pos) return agen </code></pre>

With the above-defined Agen we can render the sonification.

Different from above, let's add all rendered Agens into a sound canvas,
- using pyas .x (extend) array access
- which extends the canvas dynamically

In result we have the full sonification in a multi-channel Asig ason, which we can
- play via the Aserver.
- or e.g. for replaying at different rates (one-liner)
- or to save our result to disk (one-liner).

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code>duration = 3
# mappingspitches = pam.linlin(df.bill_length_mm.to_numpy(), 30, 60, 0, 48) + 40posses = pam.linlin(df.bill_depth_mm.to_numpy(), 10, 24, -1, 1)durs = pam.linlin(df.flipper_length_mm.to_numpy(), 0, 300, 0.1, 0.4)onsets = pam.linlin(df.body_mass_g.to_numpy(), 2800, 6500, 0, duration)
# render agens into an Asig asonason = Asig(0, channels=2, label="PMSon")for i, onset in enumerate(onsets): ason.x[{onset:None}] += asyn(pitch=pitches[i], amp=0.05, dur=durs[i], pos=posses[i]).gen_asig()
# plot and play the sonificationason.plot(offset=2,lw=1).play(onset=0.3) </code></pre>

Your browser does not support the audio element.

### 2.2. Continuous Parameter Mapping Sonification
Let's sonify environmental data from the building data set, which contains the hourly readings of temperature, humidity, solar radiation etc.


Let's prepare for modulation (continuous PMSon is mostly about modulating parameters over time).
- it is useful to turn the multivariate time series in a multi-channel Asig.
- the data is hourly for 6 months, let's choose the sampling rate so that 1s audio is 1 month of data

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code>data_asig = Asig(df, sr=24*30, cn=['temp', 'humidity', 'solar', 'wind']) # 1 month= 1sec (unit)plt.figure(); data_asig.plot(offset=2); plt.grid()plt.xlabel('time [1s=1month] before resampling')ds = data_asig.resample(44100)</code></pre>


Let' assume we want to create a PMSon which
- maps temperature to pitch of a sine tone as MIDI note range [60, 84]
- maps solar radiation to amplitude [0, 1]
- maps humidity to stereo position [left, right]
- maps wind speed to the amplitude of added white noise

We furthermore want
- to render the time interval from [2.2, 4.5] (months)
- at a rate of 0.5, i.e. 1 month in 2 seconds

Here we go: the implementation is even shorter than the specification, and readible

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code># namespace idx for more readable channel referencingidx = SimpleNamespace(**{name: i for i, name in enumerate(ds.cn)})
ds_agen = PlayAsig(ds[{2.2:4.5}], rate=0.5)agsig = SinOsc.ar(freq=ds_agen[idx.temp].linlin(0, 1, 60, 84).midi_to_cps())agpan = Pan2.ar(agsig, pos=ds_agen[idx.humidity].linlin(0,1,-1,1))agall = agpan * ds_agen[idx.solar].linlin(0, 1, 0.2, 1) + WhiteNoise() * ds_agen[idx.wind]ason = agall.gen_asig().plot(offset=2, lw=0.1).play(onset=0.5)
</code></pre>


Your browser does not support the audio element.

## 3. Earcons, Spearcons, Auditory Icons with pya AGen
### Earcons

Earcons are short motifs, i.e. sequences of pitched tones, so earcons are best composed from its scores.

Here is a rather basic earcon specification
- for simplicity using a single synth mimicking steel drums
- If needed PlayAsig can be used as Sampler on musical tones

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code>def asyn(note=60, lvl=1, dur=1, pos=0): """a simple steel drum""" f0 = pam.midi_to_cps(note) agsig = klang([ (f0, 1), (f0*3.01, 0.2), (f0*5.3, 0.1), (f0*5.13, 0.1)]) return Pan2(agsig * pam.db_to_amp(lvl) * XLine(1, 0.01, dur=dur, done="stop"), pos)
def earcon(notes, bpm=90, base_note=60, play_onset=0): """a simple earcon player for a list of note dicts""" aear = Asig(1, channels=2, label="earcon") for ev in notes: ev = SimpleNamespace(**ev) aear.x[{ev.at * 120/bpm:None}] += asyn(note=ev.note+base_note, lvl=ev.lvl, dur=ev.dur).gen_asig() return aear.play(onset=play_onset)</code></pre>

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code># an earcon playing g - c - d - ee1 = earcon(notes=( dict(at=0, note=7, dur=1.2, lvl=4, pos=-1), dict(at=0.2, note=0, dur=0.1, lvl=1, pos=-0.5), dict(at=0.3, note=2, dur=0.1, lvl=1, pos= 0.5), dict(at=0.4, note=4, dur=0.6, lvl=5, pos= 1)), bpm=90, base_note=60)
# an earcon playing g - d# - c (c-minor downward arpeggio) - 1 octave lowere2 = earcon(notes=( dict(at=0, note=7, dur=2, lvl=1, pos=0), dict(at=0.05, note=3, dur=1.5, lvl=1, pos=0), dict(at=0.1, note=0, dur=1, lvl=1, pos=0)), bpm=90, base_note=48, play_onset=2)
# to render both earcons as sequence into an audio file useearcon_sequence = e1.append(e2)earcon_sequence.plot(lw=0.5)</code></pre>


Your browser does not support the audio element.

### Auditory Icons

- Often, everyday sounds are used, e.g. crushing paper to represent file deletion.
- The ideal way to implement these is to simply load the audio files using Asig()
and play that either directly via

```PlayAsig(Asig("data/klirr.wav"), rate=0.5).gen_asig().play()```

## 4. Model-based Sonification with pya AGen

The cell below is a minimalistic implementation of the data sonogram model. It includes:
- loading of data
- feature selection / filtering
- scatter plot of data
- definition of a sonification model implementation as constrained mapping
- Mouse click interaction event anchored to the scatter plot

<pre class="github-light" style="background-color:#fff;color:#24292e;" tabindex="0"><code>fig = plt.figure(figsize=(4,4))df = penguins_dfdf.plot(x="bill_depth_mm", y="bill_length_mm", kind="scatter", s=3, ax=plt.gca());data = df.loc[:, ["bill_depth_mm", "bill_length_mm", "flipper_length_mm"]].to_numpy()
velocity = 3
def son_fn(click_pos): (WhiteNoise(200) * Line(0.6,0,0.2)**4).gen_asig().play() # initial noise
distances = np.linalg.norm(data[:,:2] - click_pos, axis=1) # distances to clickpos for i in np.argsort(distances)[:25]: # dispatch some neighbors' sounds # sonification model constraints / defines sonic parameters onset = distances[i] / velocity freq = pam.midi_to_cps(pam.linlin(data[i, 2], 150, 250, 50, 90)) amp = pam.db_to_amp(pam.linlin(distances[i], 0, 3, -6, -40)) dur = 0.8 pan = 1 if data[i,0]-click_pos[0] &gt; 0 else -1 # generate data point sound ag = SinOsc(freq) * Line(1, 0, dur)**4 Pan2(ag * amp, pan).gen_asig().stereo().play(onset=onset)
def on_click(event): if event.inaxes is not None: # if inside plot area son_fn(np.array([event.xdata, event.ydata]))fig.canvas.mpl_connect('button_press_event', on_click)</code></pre>

Your browser does not support the video element.

Notice that
- most of the steps (data handling, plotting, interaction, mapping,
parameter control) are done best in a programming environment anyway.
- the need for a sound rendering comes in at two locations in the son_def function:
- the noise signal to provide feedback that and when the model has been excited
- the data point sounds as the expanding wave reaches them (in the for loop)
- for both these cases, a 1-2 line pya-Agen specification does the job and forgoes
the need to involve an extra dependency.