voiceintelligenceresearch/MOCKS
收藏资源简介：
--- annotations_creators: - expert-generated language: - en - de - es - fr - it license: - cc-by-4.0 - mpl-2.0 multilinguality: - multilingual dataset_info: - config_name: config features: - name: audio_id dtype: string - name: audio dtype: audio: sampling_rate: 16000 - name: text dtype: string --- # MOCKS: Multilingual Open Custom Keyword Spotting Testset ## Table of Contents - [Table of Contents](#table-of-contents) - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Dataset Description - **Paper:** [MOCKS 1.0: Multilingual Open Custom Keyword Spotting Testset](https://www.isca-speech.org/archive/pdfs/interspeech_2023/pudo23_interspeech.pdf) ### Dataset Summary Multilingual Open Custom Keyword Spotting Testset (MOCKS) is a comprehensive audio testset for evaluation and benchmarking Open-Vocabulary Keyword Spotting (OV-KWS) models. It supports multiple OV-KWS problems: both text-based and audio-based keyword spotting, as well as offline and online (streaming) modes. It is based on the LibriSpeech and Mozilla Common Voice datasets and contains almost 50,000 keywords, with audio data available in English, French, German, Italian, and Spanish. The testset was generated using automatically generated alignments used for the extraction of parts of the recordings that were split into keywords and test samples. MOCKS contains both positive and negative examples selected based on phonetic transcriptions that are challenging and should allow for in-depth OV-KWS model evaluation. Please refer to our [paper](https://www.isca-speech.org/archive/pdfs/interspeech_2023/pudo23_interspeech.pdf) for further details. ### Supported Tasks and Leaderboards The MOCKS dataset can be used for the Open-Vocabulary Keyword Spotting (OV-KWS) task. It supports two OV-KWS types: - Query-by-Text, where the keyword is provided by text and needs to be detected in the audio stream. - Query-by-Example, where the keyword is provided with enrollment audio for detection in the audio stream. It also allows for: - offline keyword detection, where test audio is trimmed to contain only keywords of interest. - online (streaming) keyword detection, where test audio has past and future context besides keywords of interest. ### Languages The MOCKS incorporates 5 languages: - English - primary and largest test set, - German, - Spanish, - French, - Italian. ## Dataset Structure The MOCKS testset is split by language, source dataset, and OV-KWS type: ``` MOCKS │ └───de │ └───MCV │ │ └───test │ │ │ └───offline │ │ │ │ │ all.pair.different.tsv │ │ │ │ │ all.pair.positive.tsv │ │ │ │ │ all.pair.similar.tsv │ │ │ │ │ data.tar.gz │ │ │ │ │ subset.pair.different.tsv │ │ │ │ │ subset.pair.positive.tsv │ │ │ │ │ subset.pair.similar.tsv │ │ │ │ │ │ │ └───online │ │ │ │ │ all.pair.different.tsv │ │ │ │ │ ... │ │ │ │ data.offline.transcription.tsv │ │ │ │ data.online.transcription.tsv │ └───en │ └───LS-clean │ │ └───test │ │ │ └───offline │ │ │ │ │ all.pair.different.tsv │ │ │ │ │ ... │ │ │ │ ... │ │ │ └───LS-other │ │ └───test │ │ │ └───offline │ │ │ │ │ all.pair.different.tsv │ │ │ │ │ ... │ │ │ │ ... │ │ │ └───MCV │ │ └───test │ │ │ └───offline │ │ │ │ │ all.pair.different.tsv │ │ │ │ │ ... │ │ │ │ ... │ └───... ``` Each split is divided into: - positive examples (`all.pair.positive.tsv`) - test examples with true keywords, 5000-8000 keywords in each subset, - similar examples (`all.pair.similar.tsv`) - test examples with similar phrases to the keyword selected based on phonetic transcription distance, - different examples (`all.pair.different.tsv`) - test examples with completely different phrases. All those files contain columns separated by tab: - `keyword_path` - path to audio containing keyword phrase. - `adversary_keyword_path` - path to test audio. - `adversary_keyword_timestamp_start` - start time in seconds of phrase of interest for a given keyword from `keyword_path`, the field only available in **offline** split. - `adversary_keyword_timestamp_end` - end time in seconds of phrase of interest for a given keyword from `keyword_path`, the field only available in **offline** split. - `label` - whether the `adversary_keyword_path` contain keyword from `keyword_path` or not (1 - contains keyword, 0 - doesn't contain keyword). Each split also contains a subset of whole data with the same field structure to allow faster evaluation (`subset.pair.*.tsv`). Also, transcriptions are provided for each audio in: - `data_offline_transcription.tsv` - transcriptions for **offline** examples and `keyword_path` from **online** scenario, - `data_online_transcription.tsv` - transcriptions for the adversary, test examples from **online** scenario, three columns are present within each file: - `path_to_keyword`/`path_to_adversary_keyword` - path to the audio file, - `keyword_transcription`/`adversary_keyword_transcription` - audio transcription, - `keyword_phonetic_transcription`/`adversary_keyword_phonetic_transcription` - audio phonetic transcription. ## Using the Dataset The dataset can be used by: - downloading the archive and constructing all the test cases based on the provided `tsv` files, - `datasets` package. In the latter case, the following should work: ``` load_dataset(path="voiceintelligenceresearch/MOCKS", name="en.LS-clean", split="offline") ``` The allowed values for `name` are: - `en.LS-{clean,other}`, - `en.LS-{clean,other}.positive`, - `en.LS-{clean,other}.similar`, - `en.LS-{clean,other}.different`, - `en.LS-{clean,other}.subset`, - `en.LS-{clean,other}.positive_subset`, - `en.LS-{clean,other}.similar_subset`, - `en.LS-{clean,other}.different_subset`, - `{de,en,es,fr,it}.MCV.positive`, - `{de,en,es,fr,it}.MCV.positive.similar`, - `{de,en,es,fr,it}.MCV.positive.different`, - `{de,en,es,fr,it}.MCV.positive.subset`, - `{de,en,es,fr,it}.MCV.positive.positive_subset`, - `{de,en,es,fr,it}.MCV.positive.similar_subset`, - `{de,en,es,fr,it}.MCV.positive.different_subset`. The allowed values for `split` are: - `offline`, - `online`. `load_dataset` provides a list of the dictionary objects with the following contents: ``` { "keyword_id": datasets.Value("string"), "keyword_transcription": datasets.Value("string"), "test_id": datasets.Value("string"), "test_transcription": datasets.Value("string"), "test_audio": datasets.Audio(sampling_rate=16000), "label": datasets.Value("bool"), } ``` Each element of this list represents a single test case for the QbyT KWS: - `keyword_id` - the name of the keyword audio file in `data.tar.gz` (not used in QbyT KWS), - `keyword_transcription` - transcription of the keyword, - `test_id` - the name of the test audio file in `data.tar.gz`, - `test_transcription` - transcription of the test sample, - `test_audio` - raw data of the test audio, - `label` - `True` if the test case is positive (`keyword_transcription` is a substring of the `test_transcription`), `False` otherwise (`similar` and `different` subsets). Note that each test case can be extended to QbyE KWS by reading the proper `keyword_id` file. Unfortunately, there is no easy way to do that in the loading script. All the test files are provided in 16 kHz, even though `{de,en,es,fr,it}.MCV` files are stored in the original sampling (usually 48 kHz) in the `data.tar.gz` archives. ## Dataset Creation The MOCKS testset was created from LibriSpeech and Mozilla Common Voice (MCV) datasets that are publicly available. To create it: - a [MFA](https://mfa-models.readthedocs.io/en/latest/acoustic/index.html) with publicly available models was used to extract word-level alignments, - an internally developed, rule-based grapheme-to-phoneme (G2P) algorithm was used to prepare phonetic transcriptions for each sample. The data is stored in a 16-bit, single-channel WAV format. 16kHz sampling rate is used for LibriSpeech based testset and 48kHz sampling rate for MCV based testset. The offline testset contains an additional 0.1 seconds at the beginning and end of the extracted audio sample to mitigate the cut-speech effect. The online version contains an additional 1 second or so at the beginning and end of the extracted audio sample. The MOCKS testset is gender balanced. ## Citation Information ```bibtex @inproceedings{pudo23_interspeech, author={Mikołaj Pudo and Mateusz Wosik and Adam Cieślak and Justyna Krzywdziak and Bożena Łukasiak and Artur Janicki}, title={{MOCKS} 1.0: Multilingual Open Custom Keyword Spotting Testset}, year={2023}, booktitle={Proc. Interspeech 2023}, } ```
数据集概述
数据集名称
MOCKS: Multilingual Open Custom Keyword Spotting Testset
数据集摘要
MOCKS是一个综合的音频测试集，用于评估和基准测试开放词汇关键词识别（OV-KWS）模型。它支持多种OV-KWS问题，包括基于文本和基于音频的关键词识别，以及离线和在线（流式）模式。该数据集基于LibriSpeech和Mozilla Common Voice数据集，包含近50,000个关键词，音频数据涵盖英语、法语、德语、意大利语和西班牙语。
支持的任务和排行榜
MOCKS数据集可用于开放词汇关键词识别（OV-KWS）任务，支持两种OV-KWS类型：
- 文本查询（Query-by-Text）：通过文本提供关键词，并在音频流中检测该关键词。
- 示例查询（Query-by-Example）：通过注册音频提供关键词，并在音频流中检测该关键词。
此外，还支持：
- 离线关键词检测：测试音频被修剪以仅包含感兴趣的关键词。
- 在线（流式）关键词检测：测试音频包含关键词感兴趣的过去和未来上下文。
语言
MOCKS包含以下五种语言：
- 英语
- 德语
- 西班牙语
- 法语
- 意大利语
数据集结构
MOCKS测试集按语言、源数据集和OV-KWS类型进行划分。每个划分包含以下内容：
- 正例（
all.pair.positive.tsv）：包含真实关键词的测试示例，每个子集包含5000-8000个关键词。 - 相似例（
all.pair.similar.tsv）：包含与关键词相似的短语的测试示例，基于音素转录距离选择。 - 不同例（
all.pair.different.tsv）：包含完全不同短语的测试示例。
所有这些文件包含以下列：
keyword_path：包含关键词短语的音频路径。adversary_keyword_path：测试音频路径。adversary_keyword_timestamp_start：关键词短语的开始时间（仅在离线划分中可用）。adversary_keyword_timestamp_end：关键词短语的结束时间（仅在离线划分中可用）。label：adversary_keyword_path是否包含keyword_path中的关键词（1表示包含，0表示不包含）。
每个划分还包含一个相同字段结构的子集数据，以允许更快的评估（subset.pair.*.tsv）。
此外，每个音频的转录本也提供在：
data_offline_transcription.tsv：离线示例和在线场景中关键词路径的转录本。data_online_transcription.tsv：在线场景中对手、测试示例的转录本。
每个文件包含以下三列：
path_to_keyword/path_to_adversary_keyword：音频文件路径。keyword_transcription/adversary_keyword_transcription：音频转录本。keyword_phonetic_transcription/adversary_keyword_phonetic_transcription：音频音素转录本。
使用数据集
数据集可以通过以下方式使用：
- 下载存档并根据提供的
tsv文件构建所有测试用例。 - 使用
datasets包。
在后一种情况下，以下代码应能工作： python load_dataset(path="voiceintelligenceresearch/MOCKS", name="en.LS-clean", split="offline")
name允许的值包括：
en.LS-{clean,other}en.LS-{clean,other}.positiveen.LS-{clean,other}.similaren.LS-{clean,other}.differenten.LS-{clean,other}.subseten.LS-{clean,other}.positive_subseten.LS-{clean,other}.similar_subseten.LS-{clean,other}.different_subset{de,en,es,fr,it}.MCV.positive{de,en,es,fr,it}.MCV.positive.similar{de,en,es,fr,it}.MCV.positive.different{de,en,es,fr,it}.MCV.positive.subset{de,en,es,fr,it}.MCV.positive.positive_subset{de,en,es,fr,it}.MCV.positive.similar_subset{de,en,es,fr,it}.MCV.positive.different_subset
split允许的值包括：
offlineonline
load_dataset提供了一个字典对象列表，每个对象包含以下内容：
json
{
"keyword_id": datasets.Value("string"),
"keyword_transcription": datasets.Value("string"),
"test_id": datasets.Value("string"),
"test_transcription": datasets.Value("string"),
"test_audio": datasets.Audio(sampling_rate=16000),
"label": datasets.Value("bool"),
}
每个列表元素代表一个QbyT KWS的单个测试用例：
keyword_id：关键词音频文件在data.tar.gz中的名称（在QbyT KWS中不使用）。keyword_transcription：关键词的转录本。test_id：测试音频文件在data.tar.gz中的名称。test_transcription：测试样本的转录本。test_audio：测试音频的原始数据。label：如果测试用例为正例（keyword_transcription是test_transcription的子字符串），则为True，否则为False（similar和different子集）。
注意，每个测试用例可以通过读取适当的keyword_id文件扩展到QbyE KWS。遗憾的是，加载脚本中没有简单的方法来实现这一点。
所有测试文件以16 kHz提供，尽管{de,en,es,fr,it}.MCV文件在data.tar.gz存档中以原始采样率（通常为48 kHz）存储。
数据集创建
MOCKS测试集是从公开可用的LibriSpeech和Mozilla Common Voice（MCV）数据集创建的。创建过程如下：
- 使用公开可用的模型进行MFA（MFA）提取单词级对齐。
- 使用内部开发的基于规则的音素转录本（G2P）算法为每个样本准备音素转录本。
数据以16位、单声道WAV格式存储。LibriSpeech基于的测试集使用16 kHz采样率，MCV基于的测试集使用48 kHz采样率。
离线测试集在提取的音频样本的开始和结束处额外包含0.1秒，以减轻剪切语音效果。在线版本在提取的音频样本的开始和结束处额外包含约1秒。
MOCKS测试集是性别平衡的。
引用信息
bibtex @inproceedings{pudo23_interspeech, author={Mikołaj Pudo and Mateusz Wosik and Adam Cieślak and Justyna Krzywdziak and Bożena Łukasiak and Artur Janicki}, title={{MOCKS} 1.0: Multilingual Open Custom Keyword Spotting Testset}, year={2023}, booktitle={Proc. Interspeech 2023}, }
