Learning the sound inventory of a complex vocal skill via an intrinsic reward

Reinforcement learning (RL) is thought to underlie the acquisition of vocal skills like birdsong and speech, where sounding like one’s “tutor” is rewarding. But what RL strategy generates the rich sound inventories for song or speech? We find that the standard actor-critic model of birdsong learning fails to explain juvenile zebra finches’ efficient learning of multiple syllables. But when we replace a single actor with multiple independent actors that jointly maximize a common intrinsic reward, then birds’ empirical learning trajectories are accurately reproduced. Importantly, the influence of each actor (syllable) on the magnitude of global reward is competitively determined by its acoustic similarity to target syllables. This leads to each actor matching the target it is closest to, and occasionally, to the competitive exclusion of an actor from the learning process (i.e., the learned song). We propose that a competitive-cooperative multi-actor (MARL) algorithm is key for the efficient learning of the action inventory of a complex skill.

Methods
Part of the experimental data presented here was previously published (Lipkind et al., 2017). Source and target song models were synthetically composed of natural syllables. Harmonic syllables in the source songs were pitch-shifted by 2 or 4 semitones in the target songs using GOLDWAVE v. 5.68. Song feature calculation and clustering of syllables and calls were performed using Sound Analysis Pro. All other analyses (modes, model fitting, statistical analysis, visualization) were performed in MATLAB (Mathworks Inc).