An open dataset of multidimensional signals based on different speech patterns in pragmatic Mandarin

Speech is essential for human communication, but millions of people lose the ability to speak due to conditions such as amyotrophic lateral sclerosis (ALS) or stroke. Assistive technologies like brain-computer interfaces (BCIs), which can convert brain signals into speech, offer hope for these patients. However, these technologies still face challenges in decoding accuracy. This issue is especially challenging for tonal languages like Mandarin Chinese. Their processing requires phoneme encoding and precise tonal information handling, which complicates the decoding of brain signals. Furthermore, most existing speech datasets are based on Indo-European languages, which hinders our understanding of how tonal information is encoded in the brain. To address this, we introduce a comprehensive open dataset, which includes multimodal signals from 30 subjects using Mandarin Chinese across overt, silent, and imagined speech modes, covering electroencephalogram (EEG), surface electromyogram (sEMG), and speech recordings. Unlike many datasets that focus on a single speech mode, this one integrates three speech modes, providing a more comprehensive view of speech-related activity. Incorporating Mandarin facilitates an in-depth examination of the inner mechanisms that encode tonal variations and their interaction with motor and auditory speech representations. This is crucial for enhancing tonal language decoding in BCIs. Beyond BCI applications, this dataset lays a valuable groundwork for exploring the neural encoding of tonal languages, investigating tone-related brain dynamics, and improving assistive communication strategies. It supports cross-linguistic speech processing research and contributes to data-driven neural speech decoding technology innovations.