Multimodal Articulatory Physiological Dataset for Mandarin Chinese Based on Ultrasound Tongue Imaging

The dataset covers all the combinations of initials and finals as well as the four tones, forming 1024 complete pronunciation units. Multimodal data consists of four parts: text corpora, speech audio, lip video, and ultrasound tongue imaging, which can comprehensively reflect the physiological movement characteristics and acoustic performance during the pronunciation process. In the data quality control stage, a combination of manual verification and machine screening is adopted to eliminate invalid data such as non-standard pronunciation, blurry images, and audio distortion, ultimately obtaining a high-quality dataset. The dataset not only provides data support for basic research on the physiological mechanism of Mandarin Chinese pronunciation, the rules of tone changes, and second language acquisition, but also can be applied to practical scenarios such as speech synthesis and recognition, diagnosis and rehabilitation of speech disorders, modeling of pronunciation mechanisms, and training of artificial intelligence speech models. At the same time, it offers a reference for cross-language comparative studies on pronunciation physiology.