Multilingual-TTS

# Multilingual-TTS Gather multilingual TTS to pretrain TTS task for LLM **with total 141,718.72751177778 audio hours more than 150 languages**. ## Speech Tokenizer Convert audio to speech tokens using https://huggingface.co/neuphonic/neucodec 50Hz, **with total 23.262140693B speech tokens**. ### how to prepare speech tokens ```bash huggingface-cli download \ malaysia-ai/Multilingual-TTS \ --include "*_neucodec.zip" \ --repo-type "dataset" \ --local-dir './' huggingface-cli download --repo-type dataset \ --include 'audio_trim_neucodec-*.zip' \ --local-dir './' \ --max-workers 20 \ malaysia-ai/common_voice_22_0 hf download mesolitica/Malaysian-TTS-v2 neucodec.zip --local-dir=./ --repo-type=dataset wget https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py python3 unzip.py ``` ### how to get Neucodec json file after extracted? ```python import os def new_path(f): splitted = f.split('/') folder = f.split('/')[0] folder = folder + '_neucodec' new_f = os.path.join(folder, '/'.join(splitted[1:])) new_f = new_f.replace('.mp3', '.json').replace('.wav', '.json') return new_f token_path = new_path('AISHELL3-audio/train_wav_SSB0043_SSB00430444.mp3') print(token_path) ``` ``` AISHELL3-audio_neucodec/train_wav_SSB0043_SSB00430444.json ``` ## how to prepare audio ```bash huggingface-cli download \ malaysia-ai/Multilingual-TTS \ --include "*.zip" \ --repo-type "dataset" \ --local-dir './' huggingface-cli download --repo-type dataset \ --include '*_trim.zip' \ --local-dir './' \ --max-workers 20 \ malaysia-ai/common_voice_22_0 hf download malaysia-ai/Japanese-Single-Speaker-TTS jss_audio.zip --local-dir=./ --repo-type=dataset hf download malaysia-ai/MsceneSpeech MsceneSpeech_audio.zip --local-dir=./ --repo-type=dataset hf download malaysia-ai/Korean-Single-Speaker-TTS kss_audio.zip --local-dir=./ --repo-type=dataset wget https://gist.githubusercontent.com/huseinzol05/2e26de4f3b29d99e993b349864ab6c10/raw/9b2251f3ff958770215d70c8d82d311f82791b78/unzip.py python3 unzip.py ``` **All the audio converted to MP3 format and compressed using ZIP, we prefer ZIP instead of Parquet for easier extraction and processing later**. ## Dataset source 1. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Tamil 2. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Bengali 3. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Telugu 4. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Punjabi 5. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Malayalam 6. https://huggingface.co/datasets/shb777/gemini-flash-2.0-speech 7. https://huggingface.co/datasets/ylacombe/cml-tts 9. https://huggingface.co/datasets/parler-tts/libritts_r_filtered 10. https://huggingface.co/datasets/ylacombe/expresso 11. https://huggingface.co/datasets/yhaha/EmoVoice-DB 12. https://huggingface.co/datasets/MrDragonFox/Elise 13. https://huggingface.co/datasets/BAAI/ChildMandarin 14. https://huggingface.co/datasets/AISHELL/AISHELL-3 15. https://huggingface.co/datasets/malaysia-ai/MsceneSpeech 16. https://huggingface.co/datasets/Arsenal/StoryTTS 17. https://huggingface.co/datasets/malaysia-ai/Korean-Single-Speaker-TTS 18. https://huggingface.co/datasets/malaysia-ai/Japanese-Single-Speaker-TTS 19. https://huggingface.co/datasets/MBZUAI/ClArTTS 20. https://huggingface.co/datasets/MBZUAI/ArVoice 21. https://huggingface.co/datasets/MohamedRashad/multilingual-tts 22. https://huggingface.co/datasets/mesolitica/IMDA-TTS 23. https://huggingface.co/datasets/mesolitica/haqkiem-TTS 24. https://huggingface.co/datasets/amaai-lab/DisfluencySpeech 25. https://huggingface.co/datasets/asadshahab/maya-audio 26. https://huggingface.co/datasets/KTH/hungarian-single-speaker-tts 27. https://huggingface.co/datasets/mesolitica/Malaysian-TTS-v2 28. https://huggingface.co/datasets/thucdangvan020999/singaporean_accent_district_names_continuation 29. https://huggingface.co/datasets/thucdangvan020999/singlish-speaker2050 30. https://huggingface.co/datasets/thucdangvan020999/singlish-speaker2202 31. https://huggingface.co/datasets/aaaaaa12121212/urdu-tts-speaker3-prepared 32. https://huggingface.co/datasets/payamvha/NorthTTS 33. https://huggingface.co/datasets/yskim3271/vibravox_16k 34. https://huggingface.co/datasets/FBK-MT/Speech-MASSIVE 35. https://huggingface.co/datasets/taresh18/AnimeVox 36. https://huggingface.co/datasets/reach-vb/jenny_tts_dataset 37. https://huggingface.co/datasets/simon3000/genshin-voice 38. https://huggingface.co/datasets/MohamedRashad/Quran-Recitations 39. https://huggingface.co/datasets/ai4bharat/indicvoices_r 40. https://huggingface.co/datasets/thivux/phoaudiobook 41. https://huggingface.co/datasets/its5Q/biggest-ru-book 42. https://huggingface.co/datasets/mesolitica/nusantara-audiobook 43. https://huggingface.co/datasets/mesolitica/semisupervised-audiobook 44. https://huggingface.co/datasets/momina02/OutteTTS-urdu-dataset 45. https://huggingface.co/datasets/kresnik/zeroth_korean 46. https://huggingface.co/datasets/saeedzou/PersianVox_NM 47. https://huggingface.co/datasets/Lacito/pangloss 48. https://huggingface.co/datasets/SparkAudio/voxbox/tree/main/audios/hq-conversations 49. https://huggingface.co/datasets/SparkAudio/voxbox/tree/main/audios/magicdata 50. https://huggingface.co/datasets/Jinsaryko/Alexis 51. https://huggingface.co/datasets/thetaone-ai/HiKE <b>⁺</b> 52. https://huggingface.co/datasets/NightPrince/MasriSpeech-Full <b>⁺</b> 53. https://huggingface.co/datasets/nilc-nlp/CORAA-MUPE-ASR 54. https://huggingface.co/datasets/zeeshanparvez/andrew-v3 55. https://huggingface.co/datasets/galsenai/anta_women_tts 56. https://huggingface.co/datasets/CoRal-project/coral-v2 57. https://huggingface.co/datasets/jazza234234/david-dataset 58. https://huggingface.co/datasets/imvladikon/hebrew_speech_coursera <b>⁺</b> 59. https://huggingface.co/datasets/imvladikon/hebrew_speech_kan <b>⁺</b> 60. https://huggingface.co/datasets/nucklehead/ht-voice-dataset <b>⁺</b> 61. https://huggingface.co/datasets/DigitalUmuganda/kinyarwanda-tts-dataset 62. https://huggingface.co/datasets/AlienKevin/mixed_cantonese_and_english_speech <b>⁺</b> 63. https://huggingface.co/datasets/adnankarim/urdu_asr_data <b>⁺</b> 64. https://huggingface.co/datasets/speech-uk/voice-of-america <b>⁺</b> 65. https://huggingface.co/datasets/boniromou/zh-yue-tts-dataset 66. https://huggingface.co/datasets/MCAA1-MSU/anv_data_ke <b>⁺</b> 67. https://huggingface.co/datasets/Wenetspeech4TTS/WenetSpeech4TTS, **Premium only**. <b>⁺</b> 68. https://huggingface.co/datasets/Ken-Z/Latin-Audio <b>⁺</b> 69. https://huggingface.co/datasets/joujiboi/japanese-anime-speech-v2 <b>⁺</b> 70. https://huggingface.co/datasets/malaysia-ai/common_voice_22_0 <b>⁺</b> 71. https://huggingface.co/datasets/amphion/Emilia-NV <b>⁺</b> 72. https://huggingface.co/datasets/malaysia-ai/GTTS-Chirp3-Synthetic 73. https://huggingface.co/datasets/malaysia-ai/GTTS-WaveNet-Synthetic 74. https://huggingface.co/datasets/mesolitica/Azure-TTS-Synthetic 75. https://huggingface.co/datasets/mesolitica/GCP-TTS 76. https://huggingface.co/datasets/mesolitica/Azure-TTS-annotated 77. https://huggingface.co/datasets/Somali-tts/somali-tts-datasets <b>⁺</b> 78. https://huggingface.co/datasets/Kamtera/ParsiGoo 79. https://huggingface.co/datasets/SeyedAli/Persian-Speech-Dataset 80. https://huggingface.co/datasets/NandemoGHS/Japanese-Eroge-Voice <b>⁺</b> 81. https://huggingface.co/datasets/classla/ParlaSpeech-RS 82. https://huggingface.co/datasets/classla/ParlaSpeech-HR 83. https://huggingface.co/datasets/classla/ParlaSpeech-PL 84. https://huggingface.co/datasets/classla/ParlaSpeech-CZ 85. https://huggingface.co/datasets/tarteel-ai/everyayah 86. https://huggingface.co/datasets/imTak/korean-audio-text-develop <b>⁺</b> 87. https://huggingface.co/datasets/ylacombe/google-colombian-spanish 88. https://huggingface.co/datasets/deepghs/arknights_voices_kr 89. https://huggingface.co/datasets/deepghs/arknights_voices_jp 90. https://huggingface.co/datasets/deepghs/arknights_voices_zh 91. https://huggingface.co/datasets/deepghs/arknights_voices_en 92. https://huggingface.co/datasets/deepghs/fgo_voices_jp 93. https://huggingface.co/datasets/deepghs/azurlane_voices_jp 94. https://huggingface.co/datasets/deepghs/girlsfrontline_voices_jp 95. https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-v2 <b>⁺⁺</b> 96. https://huggingface.co/datasets/Scicom-intl/Malaysian-Chinese-Emilia <b>⁺⁺</b> 97. https://huggingface.co/datasets/naijavoices/naijavoices-dataset 98. https://huggingface.co/datasets/capleaf/viVoice <b>⁺</b> 99. https://huggingface.co/datasets/TwinkStart/KeSpeech <b>⁺</b> 100. https://huggingface.co/datasets/speech31/zeroth_korean_ipa <b>⁺</b> 101. https://huggingface.co/datasets/SparkAudio/voxbox/tree/main/audios/emilia_zh <b>⁺⁺</b> 102. https://huggingface.co/datasets/facebook/omnilingual-asr-corpus 103. https://huggingface.co/datasets/erenfazlioglu/turkishvoicedataset 104. https://huggingface.co/datasets/SparkAudio/voxbox except AISHELL-3, Emilia-CN, HQ-Conversations, MAGICDATA, WenetSpeech4TTS, Emilia-EN, EmoV-DB, Expresso, Gigaspeech, Librispeech and LibriTTS-R. 105. https://huggingface.co/datasets/mahwizzzz/UAT <b>⁺</b> 106. https://huggingface.co/datasets/Vikhrmodels/ToneWebinars <b>⁺</b> 107. https://huggingface.co/datasets/hon9kon9ize/yue_emo_speech <b>⁺</b> 108. https://huggingface.co/datasets/siddiqiya/ar-quran-hadith14books-MSA <b>⁺</b> 109. https://huggingface.co/datasets/readerbench/echo 110. https://huggingface.co/datasets/RobotsMali/afvoices <b>⁺</b> 111. https://huggingface.co/datasets/evan0617/seniortalk <b>⁺</b> 112. https://huggingface.co/datasets/skbose/indian-english-nptel-v0 113. https://huggingface.co/datasets/pnnbao-ump/VieNeu-TTS-140h 114. https://huggingface.co/datasets/inesc-id/FalAR 115. https://huggingface.co/datasets/Paradoxia/opendata-iisys-hui 116. https://huggingface.co/datasets/MohamedRashad/SADA22 <b>⁺</b> 117. https://huggingface.co/datasets/ymoslem/Wikimedia-Speech-Irish <b>⁺</b> 118. https://huggingface.co/datasets/ai4bharat/Rasa <b>⁺</b> 119. https://huggingface.co/datasets/dolly-vn/dolly-audio-1000h-vietnamese 120. https://huggingface.co/datasets/freyavoice/tr_50hrs <b>⁺</b> 121. https://huggingface.co/datasets/thennal/IMaSC 122. https://huggingface.co/datasets/alexandrainst/ftspeech 123. https://huggingface.co/datasets/ilsp/cretan-speech-corpus 124. https://huggingface.co/datasets/danielshaps/nchlt_speech_zul 125. https://huggingface.co/datasets/IqraEval/Iqra_TTS 126. https://huggingface.co/datasets/dsfsi-anv/za-african-next-voices 127. https://huggingface.co/datasets/scribe-project/npsc_nb 128. https://huggingface.co/datasets/Firoj112/highquality_nepali_female_asr 129. https://huggingface.co/datasets/ylacombe/google-tamil 130. https://huggingface.co/datasets/ylacombe/google-chilean-spanish 131. https://huggingface.co/datasets/ylacombe/google-argentinian-spanish 132. https://huggingface.co/datasets/ylacombe/google-colombian-spanish 133. https://huggingface.co/datasets/ylacombe/google-gujarati 134. https://huggingface.co/datasets/ylacombe/google-marathi 135. https://huggingface.co/datasets/tarteel-ai/EA-DI <b>⁺</b> 136. https://huggingface.co/datasets/tarteel-ai/EA-UD <b>⁺</b> 137. https://huggingface.co/datasets/parambharat/ucla_dataset <b>⁺</b> 138. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Kannada 139. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Assamese 140. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Marathi 141. https://huggingface.co/datasets/SPRINGLab/IndicTTS-Hindi 142. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Odia 143. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Manipuri 144. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Telugu 145. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Rajasthani 146. https://huggingface.co/datasets/SPRINGLab/IndicTTS-English <b>⁺</b> 147. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Marathi 148. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Gujarati 149. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Rajasthani 150. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Odia 151. https://huggingface.co/datasets/SPRINGLab/IndicTTS_Manipuri 152. https://huggingface.co/datasets/Thomcles/Persian-Farsi-Speech <b>⁺</b> 153. https://huggingface.co/datasets/Buraaq/quran-md-ayahs 154. https://huggingface.co/datasets/ivanzhu109/zh-taiwan <b>⁺</b> 155. https://huggingface.co/datasets/En1gma02/indian_accent_english <b>⁺</b> 156. https://huggingface.co/datasets/ar5entum/indic_hi_en_tts 157. https://huggingface.co/datasets/AndikaRT421/indonesian-audiobook-tts <b>⁺</b> 158. https://huggingface.co/datasets/Praha-Labs/indic-Malayalam-PD 159. https://huggingface.co/datasets/X-lord/Dataset-Text-To-Speech-Indonesia <b>⁺</b> 160. https://huggingface.co/datasets/Ritwika03/hindi_ai4bharat_indictts <b>⁺</b> 161. http://huggingface.co/datasets/evie-8/waxal-tts <b>⁺</b> 162. https://huggingface.co/datasets/asadullah797/greek-tts-dataset <b>⁺</b> 163. https://huggingface.co/datasets/MAdel121/arabic-egy-cleaned <b>⁺</b> 164. https://huggingface.co/datasets/SWivid/Habibi 165. https://huggingface.co/datasets/agufsamudra/tts-indo <b>⁺</b> 166. https://huggingface.co/datasets/NandemoGHS/Japanese-Eroge-Voice-V2 <b>⁺</b> 167. https://huggingface.co/datasets/khursanirevo/chatter 168. https://huggingface.co/datasets/Sh1man/elevenlabs 169. https://huggingface.co/datasets/MiXaiLL76/ru_book_dataset 170. https://huggingface.co/datasets/google/WaxalNLP 171. https://huggingface.co/datasets/Anilosan15/Turkish_TTS_Data 172. https://huggingface.co/datasets/Opit/bulgarian_tts 173. https://huggingface.co/datasets/speech-uk/opentts-lada <b>⁺</b> 174. https://huggingface.co/datasets/beleata74/bulgarian-audiobooks-tts-400h <b>⁺</b> 175. https://huggingface.co/datasets/datadriven-company/TTS-Romanian 176. https://huggingface.co/datasets/OrcinusOrca/YouTube-Cantonese <b>⁺</b> 177. https://huggingface.co/datasets/ARTPARK-IISc/Vaani <b>⁺</b> 178. https://huggingface.co/datasets/ziyou-li/cantonese_daily <b>⁺</b> 180. https://huggingface.co/datasets/chukypedro/africanvoices_hau 181. https://huggingface.co/datasets/openpecha/tibetan_voice <b>⁺</b> 182. https://huggingface.co/datasets/sarulab-speech/commonvoice22_sidon <b>⁺</b> 183. https://huggingface.co/datasets/ziyou-li/cantonese_daily <b>⁺</b> 184. https://huggingface.co/datasets/Appenlimited/700h-tr-turkish-text-to-speech <b>⁺</b> 185. https://huggingface.co/datasets/chukypedro/africanvoices_hau 186. https://huggingface.co/datasets/openpecha/tibetan_voice <b>⁺</b> 187. https://huggingface.co/datasets/language-and-voice-lab/samromur_children 188. https://huggingface.co/datasets/linagora/linto-dataset-audio-ar-tn <b>⁺</b> 189. https://huggingface.co/datasets/tonyshining/VLSP2020 <b>⁺</b> 190. https://huggingface.co/datasets/TingChen-ppmc/Changsha_Dialect_Conversational_Speech_Corpus 191. https://huggingface.co/datasets/damerajee/Hindi-audio-speech <b>⁺</b> 192. https://huggingface.co/datasets/doof-ferb/vlsp2020_vinai_100h <b>⁺</b> 193. https://huggingface.co/datasets/doof-ferb/VietMed_labeled 194. https://huggingface.co/datasets/sarahwei/Taiwanese-Minnan-Sutiau <b>⁺</b> 195. https://huggingface.co/datasets/Ankoholik/Enigma-Dataset <b>⁺</b> 196. https://huggingface.co/datasets/deepvk/NonverbalTTS 197. https://huggingface.co/datasets/KandirResearch/DarijaTTS-clean <b>⁺</b> 198. https://huggingface.co/datasets/shunyalabs/amharic-speech-dataset <b>⁺</b> 199. https://huggingface.co/datasets/shunyalabs/lithuanian-speech-dataset <b>⁺</b> 200. https://huggingface.co/datasets/hetchyy/everyayah-phonemes <b>⁺</b> 201. https://huggingface.co/datasets/aranemini/southern-kurdish-asr 202. https://huggingface.co/datasets/chukypedro/9jalingo-pidgin 203. https://huggingface.co/datasets/chukypedro/9jalingo-igbo 204. https://huggingface.co/datasets/chukypedro/9jalingo-hausa 205. https://huggingface.co/datasets/chukypedro/9jalingo-yoruba 206. https://huggingface.co/datasets/leyu-amharic/leyu-amharic-gojjam-dialect 207. https://huggingface.co/datasets/language-and-voice-lab/raddromur_asr <b>⁺</b> 208. https://huggingface.co/datasets/arbml/sudanese_dialect_speech <b>⁺</b> 209. https://huggingface.co/datasets/m-aliabbas/idrak_ryanspeech <b>⁺</b> 210. https://huggingface.co/datasets/ademax/vivos-vie-speech2text <b>⁺</b> 211. https://huggingface.co/datasets/dangrebenkin/sova_rudevices_audiobooks <b>⁺</b> 212. https://huggingface.co/datasets/Ranjit/or_in_dataset <b>⁺</b> 213. https://huggingface.co/datasets/HNO333333/Tibetan-0310 <b>⁺</b> 214. https://huggingface.co/datasets/AprilYang93/tibetan_wz_tts 215. https://huggingface.co/datasets/surajp/shrutilipi_sanskrit <b>⁺</b> 216. https://huggingface.co/datasets/ujs/hinglish-compressed <b>⁺</b> 217. https://huggingface.co/datasets/aaparajit02/punjabi-asr <b>⁺</b> 218. https://huggingface.co/datasets/amansingh203/stuttering_asr <b>⁺</b> 219. https://huggingface.co/datasets/ademax/vlsp-vie-speech2text1 <b>⁺</b> 220. https://huggingface.co/datasets/NbAiLab/salmon-asr-smj <b>⁺</b> 221. https://huggingface.co/datasets/joujiboi/japanese-anime-speech <b>⁺</b> 222. https://huggingface.co/datasets/imvladikon/hebrew_speech_campus <b>⁺</b> 223. https://huggingface.co/datasets/TingChen-ppmc/Tianjin_Dialect_Conversational_Speech_Corpus 224. https://huggingface.co/datasets/TingChen-ppmc/Shanghai_Dialect_Conversational_Speech_Corpus 225. https://huggingface.co/datasets/TingChen-ppmc/Zhengzhou_Dialect_Conversational_Speech_Corpus 226. https://huggingface.co/datasets/TingChen-ppmc/Changsha_Dialect_Conversational_Speech_Corpus 227. https://huggingface.co/datasets/TingChen-ppmc/Nanchang_Dialect_Conversational_Speech_Corpus 228. https://huggingface.co/datasets/yeshpanovrustem/kazakh_songs_asr <b>⁺</b> 229. https://huggingface.co/datasets/Nourhann/Arabic-Diacritized-TTS <b>⁺</b> 230. https://huggingface.co/datasets/ai4bharat/Lahaja <b>⁺</b> 231. https://huggingface.co/datasets/yeshpanovrustem/kazlibri <b>⁺</b> 232. https://huggingface.co/datasets/pnnbao-ump/ngochuyen_voice <b>⁺</b> - <b>⁺</b>, pseudospeaker, use https://huggingface.co/nvidia/speakerverification_en_titanet_large after that group using FAISS with threshold 0.1 - <b>⁺⁺</b>, pseudospeaker, from Emilia pipeline. ## Licenses Because this data repository gathers data from multiple sources, and the majority is licensed under CC BY NC 4.0, it is best to assume that everything falls under CC BY NC 4.0. ## Source code 1. Source code at https://github.com/malaysia-ai/dataset/tree/main/multilingual-tts 2. For pseudospeaker example, https://github.com/malaysia-ai/dataset/blob/main/multilingual-tts/prepare/cluster-Latin-Audio.ipynb ## Acknowledgement 1. Special thanks to [Lambda Research Grant program](https://lambda.ai/research) for Lambda cloud credit! 2. Special thanks to [Scicom-intl](https://huggingface.co/Scicom-intl) for the computation!