commonvoice22_sidon

# CV22-Sidon

## Overview
This dataset hosts a release of Mozilla Common Voice 22 restored with the Sidon speech restoration model.

- **Source**: [Mozilla Common Voice 22.0](https://commonvoice.mozilla.org/)
- **Processing**: Sidon denoising (`sarulab-speech/sidon-v0.1`) with 21 s chunks and 48 kHz reconstruction
- **Format**: [WebDataset](https://github.com/webdataset/webdataset) shards (`.tar.gz`)
- **Manifest**: `paths.yaml` enumerates every shard path for Hugging Face–style loading
- **License**: Original Common Voice license ([CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/))

## Languages
137 language folders are available; the directory names mirror the Common Voice language identifiers.

```
ab af am ar as ast az ba bas be bg bn br ca ckb cnh cs cv cy da dav de dv dyu el en eo es et eu fa fi fr fy-NL ga-IE gl gn ha he hi hsb ht hu hy-AM ia id ig is it ja ka kab kk kln kmr ko ky lg lij lo lt ltg luo lv mdf mhr mk ml mn mr mrj mt myv nan-tw nb-NO ne-NP nhi nl nn-NO nr nso oc or os pa-IN pl ps pt quy rm-sursilv rm-vallader ro ru rup rw sah sat sc sd sk skr sl sq sr st sv-SE sw ta te tg th ti tig tk tn tok tr ts tt tw ug uk ur uz ve vi vot xh yi yo yue zgh zh-CN zh-HK zh-TW zu zza
```

Each language directory contains Sidon-processed shards for the Common Voice splits that exist for that language (`train`, `validation`, `test`, `other`, and `invalidated`).

Approximate shard counts across all languages:

- `train`: 1529 shards
- `validation`: 209 shards
- `test`: 219 shards
- `other`: 1384 shards
- `invalidated`: 418 shards

## Shard Contents
Every shard stores up to 5,000 utterances. Within a shard you will find records structured as:

```
123456.flac # 48 kHz mono audio after Sidon restoration
123456.metadata.json # Original Common Voice metadata (speaker info, split ids, transcript, etc.)
```

## Usage Examples

### Hugging Face Datasets (paths manifest)
```python
import datasets
from IPython.display import Audio
from huggingface_hub import hf_hub_download
import yaml
token="YOUR_HF_ACCESS_TOKEN"
base_url = "https://huggingface.co/datasets/sarulab-speech/commonvoice22_sidon/resolve/main/"
language = 'en'
split = 'train'
data_file_path = hf_hub_download(repo_id="sarulab-speech/commonvoice22_sidon", repo_type="dataset", filename="paths.yaml",token=token)
paths = yaml.load(open(data_file_path, "r"), Loader=yaml.FullLoader)

ds = datasets.load_dataset("webdataset", data_files=[base_url + p for p in paths[language][split]],streaming=True,token=token)['train']

sample = next(iter(ds))
audio = sample['flac']
print(sample['metadata.json'])
Audio(audio['array'], rate=audio['sampling_rate'])

```

## Notes
- Sidon resamples chunks to 16 kHz internally and writes 48 kHz FLAC output.
- Failed or skipped records are omitted, so counts can differ slightly from the raw Common Voice release.

## Citation
If you use this dataset, please cite Sidon and Mozilla Common Voice.

```
@misc{nakata2025sidonfastrobustopensource,
title={Sidon: Fast and Robust Open-Source Multilingual Speech Restoration for Large-scale Dataset Cleansing},
author={Nakata, Wataru and Saito, Yuki and Ueda, Yota and Saruwatari, Hiroshi},
year={2025},
eprint={2509.17052},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2509.17052}
}
```

```
@article{ardila2020common,
author = {Ardila, Rosana and Branson, Megan and Davis, Kelly and Kohler, Michael and Meyer, Gabriela and Henretty, Reuben and Morais, Michael and Saunders, Lindsay and Tyers, Francis and Weber, Gregor},
title = {{Common Voice}: A Massively-Multilingual Speech Corpus},
journal = {Proceedings of the 12th Language Resources and Evaluation Conference (LREC)},
year = {2020}
}
```

## License
Derived from Mozilla Common Voice, this dataset remains available under [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/). Please respect any additional requirements imposed by downstream usage (e.g., speaker consent in redistribution policies).

## Acknowledgements
- Mozilla and the Common Voice contributors for collecting and releasing the original data.

# CV22-Sidon

## 概述
本数据集为经Sidon语音修复模型处理后的Mozilla Common Voice 22版本发布数据集。

- **数据源**：[Mozilla Common Voice 22.0](https://commonvoice.mozilla.org/)
- **处理流程**：采用`sarulab-speech/sidon-v0.1`实现的Sidon降噪处理，以21秒为块进行分段，并以48 kHz采样率重建音频
- **存储格式**：采用[WebDataset](https://github.com/webdataset/webdataset)格式的分片文件（`.tar.gz`）
- **索引文件**：`paths.yaml` 枚举了所有分片路径，支持Hugging Face风格的数据集加载
- **授权协议**：沿用原始Common Voice的授权协议（[CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/)）

## 语言支持
本数据集包含137个语言文件夹，目录名称与Common Voice的语言标识符保持一致。

ab af am ar as ast az ba bas be bg bn br ca ckb cnh cs cv cy da dav de dv dyu el en eo es et eu fa fi fr fy-NL ga-IE gl gn ha he hi hsb ht hu hy-AM ia id ig is it ja ka kab kk kln kmr ko ky lg lij lo lt ltg luo lv mdf mhr mk ml mn mr mrj mt myv nan-tw nb-NO ne-NP nhi nl nn-NO nr nso oc or os pa-IN pl ps pt quy rm-sursilv rm-vallader ro ru rup rw sah sat sc sd sk skr sl sq sr st sv-SE sw ta te tg th ti tig tk tn tok tr ts tt tw ug uk ur uz ve vi vot xh yi yo yue zgh zh-CN zh-HK zh-TW zu zza

每个语言目录均包含对应语言下Common Voice现有数据集划分的Sidon处理分片，涵盖`train`、`validation`、`test`、`other`及`invalidated`划分。

全语言分片数量近似如下：
- `train`：1529个分片
- `validation`：209个分片
- `test`：219个分片
- `other`：1384个分片
- `invalidated`：418个分片

## 分片内容
每个分片最多存储5000条语音样本。分片内的文件结构如下：

123456.flac # 经Sidon修复后的48 kHz单声道音频文件
123456.metadata.json # 原始Common Voice元数据，包含说话人信息、数据集划分ID、转录文本等内容

## 使用示例

### Hugging Face 数据集（索引文件方式）
python
import datasets
from IPython.display import Audio
from huggingface_hub import hf_hub_download
import yaml
token="YOUR_HF_ACCESS_TOKEN"
base_url = "https://huggingface.co/datasets/sarulab-speech/commonvoice22_sidon/resolve/main/"
language = 'en'
split = 'train'
data_file_path = hf_hub_download(repo_id="sarulab-speech/commonvoice22_sidon", repo_type="dataset", filename="paths.yaml",token=token)
paths = yaml.load(open(data_file_path, "r"), Loader=yaml.FullLoader)

ds = datasets.load_dataset("webdataset", data_files=[base_url + p for p in paths[language][split]],streaming=True,token=token)['train']

sample = next(iter(ds))
audio = sample['flac']
print(sample['metadata.json'])
Audio(audio['array'], rate=audio['sampling_rate'])

## 注意事项
- Sidon内部会将语音块重采样至16 kHz，最终输出48 kHz采样率的FLAC音频文件。
- 处理失败或被跳过的语音样本将被剔除，因此本数据集的样本数量与原始Common Voice发布版本可能存在细微差异。

## 引用说明
若您使用本数据集，请同时引用Sidon与Mozilla Common Voice的相关文献。

@misc{nakata2025sidonfastrobustopensource,
title={Sidon: Fast and Robust Open-Source Multilingual Speech Restoration for Large-scale Dataset Cleansing},
author={Nakata, Wataru and Saito, Yuki and Ueda, Yota and Saruwatari, Hiroshi},
year={2025},
eprint={2509.17052},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2509.17052}
}

@article{ardila2020common,
author = {Ardila, Rosana and Branson, Megan and Davis, Kelly and Kohler, Michael and Meyer, Gabriela and Henretty, Reuben and Morais, Michael and Saunders, Lindsay and Tyers, Francis and Weber, Gregor},
title = {{Common Voice}: A Massively-Multilingual Speech Corpus},
journal = {Proceedings of the 12th Language Resources and Evaluation Conference (LREC)},
year = {2020}
}

## 授权协议
本数据集衍生自Mozilla Common Voice，仍采用[CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/)协议进行分发。请遵守下游使用场景下的额外要求（例如，再分发时需获得说话人同意）。

## 致谢
- Mozilla及Common Voice项目贡献者收集并发布原始数据集。