lab260/espeech_balalaika

--- license: apache-2.0 task_categories: - text-to-speech - automatic-speech-recognition language: - ru pretty_name: ESpeech datasets annotate by Balalaika --- # ESpeech datasets (w/o podcasts) Annotated by Balalaika **A curated Russian speech dataset for advanced speech generative tasks.** ## Overview **ESpeech datasets Annotated by Balalaika** is a high-quality Russian speech corpus, meticulously filtered and annotated by the **lab260 team at MTUCI** with the latest version of our pipeline, **BALALAIKA**. - **Language:** Russian only - **Genres:** Audiobooks - **Source:** ESpeech datasets ([HF link](https://huggingface.co/ESpeech datasets)) - **License:** apache-2.0 - **Total Duration After Filtering:** 475.892 from 1726 - **Format:** Parquet files with split-wise annotation *** ## Usage **Primary Use Cases:** - Text-to-Speech (TTS) generation - Automatic Speech Recognition (ASR) - Analysis of accent, stress, and prosody - Russian speech technology research ### 1. Download the dataset ### 2. Extract the files ```basg for archive in *.tar.gz; do dir="${archive%.tar.gz}" mkdir -p "$dir" tar -xzvf "$archive" -C "$dir" rm "$archive" done ``` ### 3. Load data in PyTorch ```python from pathlib import Path import pandas as pd from torch.utils.data import Dataset import torchaudio class ParquetConcatDataset(Dataset): def __init__(self, parquet_dir, audio_root, parse_fn=None): self.parquet_dir = Path(parquet_dir) self.audio_root = Path(audio_root) parquet_files = list(self.parquet_dir.glob("*.parquet")) dfs = [pd.read_parquet(f) for f in parquet_files] self.df = pd.concat(dfs, ignore_index=True) def __len__(self): return len(self.df) def __getitem__(self, idx): row = self.df.iloc[idx] audio_path = self.audio_root / row["filepath"] waveform, sample_rate = torchaudio.load(audio_path) return { "audio_path": str(audio_path), "waveform": waveform, "sample_rate": sample_rate, "nisqa_mos": row["MOS"], "nisqa_noi": row["NOI"], "nisqa_dis": row["DISC"], "nisqa_col": row["COL"], "nisqa_loud": row["LOUD"], "is_single_speaker": bool(row["is_single_speaker"]), "accented_text": row["accent"], "asr_text": row["rover"], "punctuated_text": row["punct"] } # Example usage ds = ParquetConcatDataset( PATH_TO_PARQUETS_DIR, PATH_TO_AUDIO_ROOT ) ``` `PATH_TO_PARQUETS_DIR`: Path to the folder containing all .parquet files with metadata and annotations for the dataset. `PATH_TO_AUDIO_ROOT`: Path to the root directory containing all audio subfolders and files referenced by filepath columns in the metadata. *** ## Data Processing & Annotation Our pipeline applies **rigorous filtering and enrichment** steps: 1. **Removed speech segments** shorter than 1 seconds 2. **Filtered segments** with [NISQA](https://github.com/gabrielmittag/NISQA/tree/master/nisqa) MOS < 4.0 for quality assurance 3. **Excluded segments with multiple speakers** (via [pyannotate diarization](https://huggingface.co/pyannote/speaker-diarization-community-1)) 4. **Filtered out speech with music background** (custom music detector) 5. **Revised transcriptions:** Crowd-sourced with multiple ASRs, fused via ROVER ([T-one](https://github.com/voicekit-team/T-one/tree/main), [GigaAMv3-rnnt, GigaAMv3-ctc, GigaAMv3-ctc-lm](https://github.com/salute-developers/GigaAM), [vosk](https://huggingface.co/alphacep/vosk-model-ru)) 6. **Punctuation added** using [RuPunct](https://huggingface.co/RUPunct/RUPunct_big) 7. **Stress marks added** via [RuAccent](https://github.com/Den4ikAI/ruaccent) 8. **IPA phonemization** performed with our own neural model All **annotation fields** are handled and provided separately for transparency and flexibility. *** ## Data Structure - **Annotation storage:** Parquet files - **Speech storage:** .tar.gz files with speech segments in .mp3 - **Annotations:** Each sample includes separate fields for: - **Filepath** - **Quality metrics: MOS, NOI, DIS, COL, LOUD** - **Model for quality assesment** - **Transcript with stresses and pucntuation** - **Transcript after ROVER** - **Transcript with punctuation** - **IPA transcription** - **Speaker diarization flag** *** ## How to Cite Please cite the following paper if you use this dataset in research: ``` @misc{borodin2025datacentricframeworkaddressingphonetic, title={A Data-Centric Framework for Addressing Phonetic and Prosodic Challenges in Russian Speech Generative Models}, author={Kirill Borodin and Nikita Vasiliev and Vasiliy Kudryavtsev and Maxim Maslov and Mikhail Gorodnichev and Oleg Rogov and Grach Mkrtchian}, year={2025}, eprint={2507.13563}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2507.13563}, } ``` *** ## Contact - Telegram: [@korallll_ai](https://t.me/korallll_ai) - Email: [[e-mail]](mailto:[e-mail]) *** ## Links - [Balalaka annotation pipeline](https://github.com/mtuciru/balalaika/tree/main/src) - [Other datasets annotated by BALALAIKA](https://huggingface.co/collections/MTUCI/balalaika-dataset) - [Custom models' inference implementaton](https://huggingface.co/collections/MTUCI/balalaika-models) - [Paper (arXiv)](https://arxiv.org/pdf/2507.13563) - [OpenSTT repository](https://github.com/snakers4/open_stt?tab=readme-ov-file) - [NISQA](https://github.com/gabrielmittag/NISQA/tree/master/nisqa) - [pyannotate diarization](https://huggingface.co/pyannote/speaker-diarization-community-1) - [T-one](https://github.com/voicekit-team/T-one/tree/main) - [GigaAM v3-rnnt, GigaAMv3-ctc, GigaAMv3-ctc-lm](https://github.com/salute-developers/GigaAM) - [vosk](https://huggingface.co/alphacep/vosk-model-ru) - [RuPunct](https://huggingface.co/RUPunct/RUPunct_big) - [RuAccent](https://github.com/Den4ikAI/ruaccent) *** ## License Distributed under **Apache 2.0** ***--- license: apache-2.0 task_categories: - 文本转语音（Text-to-Speech, TTS） - 自动语音识别（Automatic Speech Recognition, ASR） language: - 俄语（ru） pretty_name: Balalaika标注的ESpeech数据集 --- # 由Balalaika标注的ESpeech数据集（不含播客） **专为高级语音生成任务打造的精选俄语语音数据集。** ## 概览 **由Balalaika标注的ESpeech数据集**是高质量俄语语音语料库，由MTUCI的lab260团队使用我们最新版本的标注流水线**BALALAIKA**精心筛选并标注。 - **语言：** 仅俄语 - **语料体裁：** 有声书 - **数据来源：** ESpeech数据集（[Hugging Face链接](https://huggingface.co/ESpeech datasets)） - **许可证：** Apache 2.0 - **过滤后总时长：** 475.892小时，源自1726条原始数据 - **数据格式：** 带分块标注的Parquet格式文件 *** ## 使用方法 **核心应用场景：** - 文本转语音（TTS）生成 - 自动语音识别（ASR） - 口音、重音与韵律分析 - 俄语语音技术研究 ### 1. 下载数据集 ### 2. 解压文件 bash for archive in *.tar.gz; do dir="${archive%.tar.gz}" mkdir -p "$dir" tar -xzvf "$archive" -C "$dir" rm "$archive" done ### 3. 在PyTorch中加载数据 python from pathlib import Path import pandas as pd from torch.utils.data import Dataset import torchaudio class ParquetConcatDataset(Dataset): def __init__(self, parquet_dir, audio_root, parse_fn=None): self.parquet_dir = Path(parquet_dir) self.audio_root = Path(audio_root) parquet_files = list(self.parquet_dir.glob("*.parquet")) dfs = [pd.read_parquet(f) for f in parquet_files] self.df = pd.concat(dfs, ignore_index=True) def __len__(self): return len(self.df) def __getitem__(self, idx): row = self.df.iloc[idx] audio_path = self.audio_root / row["filepath"] waveform, sample_rate = torchaudio.load(audio_path) return { "audio_path": str(audio_path), "waveform": waveform, "sample_rate": sample_rate, "nisqa_mos": row["MOS"], "nisqa_noi": row["NOI"], "nisqa_dis": row["DISC"], "nisqa_col": row["COL"], "nisqa_loud": row["LOUD"], "is_single_speaker": bool(row["is_single_speaker"]), "accented_text": row["accent"], "asr_text": row["rover"], "punctuated_text": row["punct"] } # Example usage ds = ParquetConcatDataset( PATH_TO_PARQUETS_DIR, PATH_TO_AUDIO_ROOT ) `PATH_TO_PARQUETS_DIR`：指向包含数据集所有元数据与标注的Parquet文件的文件夹路径。 `PATH_TO_AUDIO_ROOT`：指向包含元数据中`filepath`字段所引用的所有音频子文件夹与音频文件的根目录路径。 *** ## 数据处理与标注 我们的流水线采用了**严格的筛选与增强**步骤： 1. 移除时长不足1秒的语音片段 2. 为保证质量，过滤掉[NISQA](https://github.com/gabrielmittag/NISQA/tree/master/nisqa)平均意见得分（MOS）低于4.0的片段 3. 排除包含多位说话人的片段（通过[pyannotate 说话人 diarization](https://huggingface.co/pyannote/speaker-diarization-community-1)工具） 4. 过滤掉带有背景音乐的语音（自研音乐检测器） 5. 转录文本修订：通过众包结合多种自动语音识别（ASR）结果，使用ROVER算法进行融合（使用的模型包括[T-one](https://github.com/voicekit-team/T-one/tree/main)、[GigaAMv3-rnnt、GigaAMv3-ctc、GigaAMv3-ctc-lm](https://github.com/salute-developers/GigaAM)以及[vosk](https://huggingface.co/alphacep/vosk-model-ru)） 6. 使用[RuPunct](https://huggingface.co/RUPunct/RUPunct_big)为文本添加标点符号 7. 通过[RuAccent](https://github.com/Den4ikAI/ruaccent)为文本添加重音标记 8. 使用自研神经模型进行国际音标（International Phonetic Alphabet, IPA）音素转写 所有**标注字段**均单独提供，以保证透明度与使用灵活性。 *** ## 数据结构 - **标注存储：** Parquet格式文件 - **语音存储：** 以.tar.gz压缩包形式存储的.mp3格式语音片段 - **标注内容：** 每条样本包含以下独立字段： - **文件路径** - **质量指标：** 平均意见得分（MOS）、噪声评分（NOI）、失真评分（DISC）、色彩失真评分（COL）与响度评分（LOUD） - **质量评估模型** - **带重音与标点的转录文本** - **ROVER融合后的转录文本** - **仅带标点的转录文本** - **国际音标（IPA）转写文本** - **单说话人标记** *** ## 引用方式 如果您在研究中使用本数据集，请引用以下论文： bibtex @misc{borodin2025datacentricframeworkaddressingphonetic, title={A Data-Centric Framework for Addressing Phonetic and Prosodic Challenges in Russian Speech Generative Models}, author={Kirill Borodin and Nikita Vasiliev and Vasiliy Kudryavtsev and Maxim Maslov and Mikhail Gorodnichev and Oleg Rogov and Grach Mkrtchian}, year={2025}, eprint={2507.13563}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2507.13563}, } *** ## 联系方式 - Telegram: [@korallll_ai](https://t.me/korallll_ai) - 电子邮箱: [[e-mail]](mailto:[e-mail]) *** ## 相关链接 - [Balalaika标注流水线](https://github.com/mtuciru/balalaika/tree/main/src) - [其他由BALALAIKA标注的数据集](https://huggingface.co/collections/MTUCI/balalaika-dataset) - [自研模型推理实现](https://huggingface.co/collections/MTUCI/balalaika-models) - [论文（arXiv）](https://arxiv.org/pdf/2507.13563) - [OpenSTT代码仓库](https://github.com/snakers4/open_stt?tab=readme-ov-file) - [NISQA](https://github.com/gabrielmittag/NISQA/tree/master/nisqa) - [pyannotate 说话人 diarization工具](https://huggingface.co/pyannote/speaker-diarization-community-1) - [T-one](https://github.com/voicekit-team/T-one/tree/main) - [GigaAM v3-rnnt、GigaAMv3-ctc与GigaAMv3-ctc-lm](https://github.com/salute-developers/GigaAM) - [vosk](https://huggingface.co/alphacep/vosk-model-ru) - [RuPunct](https://huggingface.co/RUPunct/RUPunct_big) - [RuAccent](https://github.com/Den4ikAI/ruaccent) *** ## 许可证 本数据集采用**Apache 2.0**许可证分发。