Data for: An instantaneous voice synthesis neuroprosthesis

Brain computer interfaces (BCIs) have the potential to restore
communication for people who have lost the ability to speak due to
neurological disease or injury. BCIs have been used to translate the
neural correlates of attempted speech into text. However, text
communication fails to capture the nuances of human speech such as prosody
and immediately hearing one’s own voice. Here, we demonstrate a
“brain-to-voice” neuroprosthesis that instantaneously synthesizes voice
with closed-loop audio feedback by decoding neural activity from 256
microelectrodes implanted into the ventral precentral gyrus of a man with
amyotrophic lateral sclerosis and severe dysarthria. We overcame the
challenge of lacking ground-truth speech for training the neural decoder
and were able to accurately synthesize his voice. Along with phonemic
content, we were also able to decode paralinguistic features from
intracortical activity, enabling the participant to modulate his
BCI-synthesized voice in real-time to change intonation and sing short
melodies. These results demonstrate the feasibility of enabling people
with paralysis to speak intelligibly and expressively through a BCI. The
neural data associated with this study is available here.