ChineseEEG-2:An EEG Dataset for Multimodal Semantic Alignment and Neural Decoding during Reading and Listening

Now-existing Electroencephalography (EEG) datasets are mainly based on English, which encounters difficulty when representing Chinese. While there have been EEG datasets related to linguistic stimuli, the existing resources are limited, and many faces the problem of teacher-forcing. In our future studies, we plan to promote a unified encoder of multi-modalities for semantic decoding, which suggests the need of more data support. To bridge this gap, we introduce ChineseEEG-2, a high-density EEG dataset that extends ChineseEEG containing both reading aloud and auditory listening tasks. As a unique multimodal EEG dataset featuring synchronized reading and listening tasks based on the same corpus, ChineseEEG-2 dataset enables the exploration of how the brain processes language across both visual and auditory modalities in the context of Chinese natural language. It offers valuable insights into multimodal semantic alignment, neural decoding, and the alignment between large language models and neural processes, contributing to the development of BCI systems for language decodingA total of 12 healthy participants were recruited for the study, ranging in age from 18 to 25 years (mean age: 21.9 years; 4 males, 8 females). Among the 12 participants, four (2 males and 2 females) conducted the reading task, while the rest eight conducted the passive listening task. In the formal experiment, in total, 10.8 hours of reading data (around 3 hours per subject) and approximately 21.6 hours of listening data (around 3 hours per subject) were collected, amounting to 32.4 hours of data overall.*Note: To protect participant privacy, the original audio recordings are not available, as the human voice is a biometric identifier. Instead, we provide the audio embeddings in the materials&embeddings folder for your analysis.