FluidInference/cv-corpus-25.0-ja
收藏资源简介：
--- language: - ja license: cc0-1.0 task_categories: - automatic-speech-recognition pretty_name: Mozilla Common Voice 25.0 - Japanese Test Set size_categories: - 1K<n<10K tags: - speech - audio - japanese - asr - common-voice - test-set --- # Mozilla Common Voice 25.0 - Japanese Test Set (Complete) ## Dataset Description Complete Japanese test set from Mozilla Common Voice Corpus 25.0. This dataset contains **all 9,019 validated test samples**, compared to the partial 2,334-sample version previously available on HuggingFace. ### Key Features - **Size**: 9,019 validated test utterances - **Coverage**: 100% of official Common Voice 25.0 Japanese test split - **Multi-speaker**: Diverse set of speakers with demographic metadata - **Quality**: Community-validated recordings - **Format**: MP3 audio files with full metadata - **Use case**: Standard test set for Japanese ASR evaluation ## Why This Dataset? The previous HuggingFace repository (FluidInference/cv-corpus-25.0-ja) only contained **2,334 test files (26%)** due to incomplete uploads. This dataset provides: - ✅ **All 9,019 test files (100%)** - ✅ Complete metadata matching official Mozilla release - ✅ Ready-to-use format for ASR benchmarking - ✅ No missing files or metadata mismatches ## Dataset Structure ### Files ``` cv-corpus-25.0-ja-test-only/ ├── manifest.json # Dataset manifest with split information ├── load_dataset.py # Helper script to load all splits ├── ja_00/ │ ├── clips/ # 3,000 MP3 files │ └── test.jsonl # Metadata for this split ├── ja_01/ │ ├── clips/ # 3,000 MP3 files │ └── test.jsonl # Metadata for this split ├── ja_02/ │ ├── clips/ # 3,000 MP3 files │ └── test.jsonl # Metadata for this split └── ja_03/ ├── clips/ # 19 MP3 files └── test.jsonl # Metadata for this split ``` ### Metadata Format Each `test.jsonl` file contains entries like: ```json { "file_name": "common_voice_ja_12345.mp3", "path": "clips/common_voice_ja_12345.mp3", "text": "Japanese transcription text", "client_id": "anonymous_speaker_id", "up_votes": 2, "down_votes": 0, "age": "thirties", "gender": "male", "accent": "", "locale": "ja" } ``` **Note**: Files are split across 4 directories (ja_00, ja_01, ja_02, ja_03) to comply with HuggingFace's 10,000 files per directory limit. Each directory contains a subset of the full test set. ## Dataset Manifest The `manifest.json` file provides complete information about the dataset structure: ```json { "dataset_name": "Mozilla Common Voice 25.0 - Japanese Test Set", "total_samples": 9019, "total_size_mb": 247.16, "num_splits": 4, "splits": [ {"name": "ja_00", "num_files": 3000, ...}, {"name": "ja_01", "num_files": 3000, ...}, {"name": "ja_02", "num_files": 3000, ...}, {"name": "ja_03", "num_files": 19, ...} ] } ``` ## Usage ### Loading with Python ```python import json from pathlib import Path def load_cv_test_set(dataset_dir="."): dataset_dir = Path(dataset_dir) samples = [] # Load from all splits (ja_00, ja_01, ja_02, ja_03) for split_dir in sorted(dataset_dir.glob("ja_[0-9][0-9]")): metadata_file = split_dir / "test.jsonl" with open(metadata_file, 'r', encoding='utf-8') as f: for line in f: entry = json.loads(line) entry['audio_path'] = str(split_dir / entry['path']) samples.append(entry) return samples # Load complete test set test_samples = load_cv_test_set() print(f"Loaded {len(test_samples)} test samples") ``` ### ASR Benchmarking ```python # Evaluate your ASR model for sample in test_samples: audio_path = sample['audio_path'] reference = sample['text'] # Your ASR inference here hypothesis = your_asr_model(audio_path) # Calculate CER/WER cer = calculate_cer(reference, hypothesis) ``` ## Dataset Statistics - **Total samples**: 9,019 - **Language**: Japanese (ja) - **Format**: MP3 (various bitrates) - **License**: CC0 1.0 (Public Domain) - **Source**: Mozilla Common Voice 25.0 - **Split**: Test only ## Comparison with Other Datasets | Dataset | Samples | Completeness | |---------|---------|--------------| | FluidInference/cv-corpus-25.0-ja | 2,334 | 26% | | **This dataset** | **9,019** | **100%** | | Original Mozilla CV 25.0 test | 9,019 | 100% | ## Citation If you use this dataset, please cite the original Common Voice project: ```bibtex @inproceedings{commonvoice:2020, author = {Ardila, R. and Branson, M. and Davis, K. and Henretty, M. and Kohler, M. and Meyer, J. and Morais, R. and Saunders, L. and Tyers, F. M. and Weber, G.}, title = {Common Voice: A Massively-Multilingual Speech Corpus}, booktitle = {Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020)}, pages = {4211--4215}, year = {2020} } ``` ## License CC0 1.0 Universal (Public Domain) The Common Voice dataset is released under CC0, meaning you can: - Use for any purpose (commercial or non-commercial) - Modify and redistribute - No attribution required (though appreciated) ## Original Source - **Project**: [Mozilla Common Voice](https://commonvoice.mozilla.org/) - **Version**: 25.0 (released 2026-03-09) - **Language**: Japanese (ja) - **Original download**: https://commonvoice.mozilla.org/ja/datasets ## Dataset Quality All samples in this dataset have been: - ✅ Validated by community members - ✅ Checked for audio quality - ✅ Verified for transcription accuracy - ✅ Filtered from invalidated/reported samples ## Use Cases - Japanese ASR model evaluation - Benchmarking speech recognition systems - Speaker diversity analysis - Accent and demographic studies - Standard test set for research papers ## Acknowledgments Thanks to the Mozilla Common Voice community and all the contributors who recorded and validated these audio samples.
