realnetworks-kontxt/fleurs-hs-vits

Name: realnetworks-kontxt/fleurs-hs-vits
Creator: realnetworks-kontxt
Published: 2024-06-11T22:53:47+08:00
License: https://creativecommons.org/licenses/by/4.0/

Hugging Face2024-12-19 更新2024-06-11 收录

合成语音检测

多语言

数据链接：

https://hf-mirror.com/datasets/realnetworks-kontxt/fleurs-hs-vits 数据链接链接失效反馈

官方服务：

资源简介：

--- license: cc-by-4.0 task_categories: - audio-classification language: - de - en - es - fr - it - nl - pl - sv tags: - speech - speech-classifiation - text-to-speech - spoofing - multilingualism pretty_name: FLEURS-HS VITS size_categories: - 10K<n<100K --- # FLEURS-HS VITS An extension of the [FLEURS](https://huggingface.co/datasets/google/fleurs) dataset for synthetic speech detection using text-to-speech, featured in the paper **Synthetic speech detection with Wav2Vec 2.0 in various language settings**. This dataset is 1 of 3 used in the paper, the others being: - [FLEURS-HS](https://huggingface.co/datasets/realnetworks-kontxt/fleurs-hs) - the default train, dev and test sets - separated due to different licensing - [ARCTIC-HS](https://huggingface.co/datasets/realnetworks-kontxt/arctic-hs) - extension of the [CMU_ARCTIC](http://festvox.org/cmu_arctic/) and [L2-ARCTIC](https://psi.engr.tamu.edu/l2-arctic-corpus/) sets in a similar manner ## Dataset Details ### Dataset Description The dataset features 8 languages originally seen in FLEURS: - German - English - Spanish - French - Italian - Dutch - Polish - Swedish The `synthetic` samples are generated using: - [Google Cloud Text-To-Speech](https://cloud.google.com/text-to-speech) - [Azure Text-To-Speech](https://azure.microsoft.com/en-us/products/ai-services/text-to-speech) - [Amazon Polly](https://aws.amazon.com/polly/) Only the test VITS samples are provided. For every VITS voice, which is in practice specific model weights, one sample per transcript is provided. - **Curated by:** [KONTXT by RealNetworks](https://realnetworks.com/kontxt) - **Funded by:** [RealNetworks](https://realnetworks.com/) - **Language(s) (NLP):** English, German, Spanish, French, Italian, Dutch, Polish, Swedish - **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) for the code, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) for the dataset (but various licenses depending on the source for VITS samples) ### Dataset Sources The original FLEURS dataset was downloaded from [HuggingFace](https://huggingface.co/datasets/google/fleurs). - **FLEURS Repository:** [HuggingFace](https://huggingface.co/datasets/google/fleurs) - **FLEURS Paper:** [arXiv](https://arxiv.org/abs/2205.12446) - **Paper:** Synthetic speech detection with Wav2Vec 2.0 in various language settings ## Uses This dataset is best used as a difficult test set. Each sample contains an `Audio` feature, and a label, which is always `synthetic`; this dataset does not include any human samples. ### Direct Use The following snippet of code demonstrates loading the training split for English: ```python from datasets import load_dataset fleurs_hs = load_dataset( "realnetworks-kontxt/fleurs-hs-vits", "en_us", split="test", trust_remote_code=True, ) ``` To load a different language, change `en_us` into one of the following: - `de_de` for German - `es_419` for Spanish - `fr_fr` for French - `it_it` for Italian - `nl_nl` for Dutch - `pl_pl` for Polish - `sv_se` for Swedish This dataset only has a `test` split. The `trust_remote_code=True` parameter is necessary because this dataset uses a custom loader. To check out which code is being ran, check out the [loading script](./fleurs-hs-vits.py). ## Dataset Structure The dataset data is contained in the [data directory](https://huggingface.co/datasets/realnetworks-kontxt/fleurs-hs-vits/tree/main/data). There exists 1 directory per language. Within that directory, there is a directory named `splits`; it contains 1 file per split: - `test.tar.gz` That `.tar.gz` file contains 1 or more directories, named after the VITS model being used: ex. `thorsten-vits` Each of these directories contain `.wav` files. Each `.wav` file is named after the ID of its transcript. Keep in mind that these directories can't be merged as they share their file names. An identical file name implies a speaker-voice pair, ex. `human/123.wav` and `thorsten-vits/123.wav`. Finally, back to the language directory, it contains 3 metadata files, which are not used in the loaded dataset, but might be useful to researchers: - `recording-metadata.csv` - contains the transcript ID, file name, split and gender of the original FLEURS samples - `recording-transcripts.csv` - contains the transcrpits of the original FLEURS samples - `voice-metadata.csv` - contains the groupation of TTS' used alongside the splits they were used for ### Sample A sample contains contains an Audio feature `audio`, and a string `label`. ``` { 'audio': { 'path': 'ljspeech-vits/1660.wav', 'array': array([0.00119019, 0.00109863, 0.00106812, ..., 0., 0., 0.]), 'sampling_rate': 16000 }, 'label': 'synthetic' } ``` ## Citation The dataset is featured alongside our paper, **Synthetic speech detection with Wav2Vec 2.0 in various language settings**, which will be published on IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). We'll provide links once it's available online. **BibTeX:** If you use this work, please cite us by including the following BibTeX reference: ``` @inproceedings{dropuljic-ssdww2v2ivls, author={Dropuljić, Branimir and Šuflaj, Miljenko and Jertec, Andrej and Obadić, Leo}, booktitle={{IEEE} International Conference on Acoustics, Speech, and Signal Processing, {ICASSP} 2024 - Workshops, Seoul, Republic of Korea, April 14-19, 2024}, title={Synthetic Speech Detection with Wav2vec 2.0 in Various Language Settings}, year={2024}, month={04}, pages={585-589}, publisher={{IEEE}}, volume={}, number={}, keywords={Synthetic speech detection;text-to-speech;wav2vec 2.0;spoofing attack;multilingualism}, url={https://doi.org/10.1109/ICASSPW62465.2024.10627750}, doi={10.1109/ICASSPW62465.2024.10627750} } ``` ## Dataset Card Authors - [Miljenko Šuflaj](https://huggingface.co/suflaj) ## Dataset Card Contact - [Miljenko Šuflaj](mailto:msuflaj@realnetworks.com)

提供机构：

realnetworks-kontxt

原始信息汇总

FLEURS-HS VITS 数据集概述

数据集基本信息

名称: FLEURS-HS VITS
任务类别: 音频分类
语言: 德语, 英语, 西班牙语, 法语, 意大利语, 荷兰语, 波兰语, 瑞典语
标签: 语音, 语音分类, 文本到语音, 欺骗, 多语言
数据集大小: 10K<n<100K
许可证: 数据集为CC BY 4.0，代码为Apache 2.0

数据集描述

内容: 该数据集是FLEURS数据集的扩展，用于合成语音检测，包含8种语言的合成语音样本。
合成语音生成工具: Google Cloud Text-To-Speech, Azure Text-To-Speech, Amazon Polly
数据集构成: 仅包含测试样本，每个VITS声音模型对应一个样本。

数据集结构

目录结构: 每个语言一个目录，内部包含splits目录，存放test.tar.gz文件，该文件包含多个以VITS模型命名的目录，每个目录下是.wav文件。
元数据文件: 包含recording-metadata.csv, recording-transcripts.csv, voice-metadata.csv，提供原始样本的元数据信息。

使用方法

加载示例: 使用datasets.load_dataset函数加载特定语言的测试集。
数据集分割: 仅有一个test分割。

引用信息

相关论文: Synthetic speech detection with Wav2Vec 2.0 in various language settings
会议: IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)
BibTeX: 待更新

数据集联系人

作者: Miljenko Šuflaj
联系方式: msuflaj@realnetworks.com

搜集汇总

数据集介绍

realnetworks-kontxt/fleurs-hs-vits 数据集图片

构建方式

FLEURS-HS VITS 数据集是基于 FLEURS 语料库的扩展，专门用于合成语音检测任务。构建过程中，研究人员利用三种商业文本转语音服务——Google Cloud Text-To-Speech、Azure Text-To-Speech 和 Amazon Polly——生成合成语音样本。每个 VITS 模型权重对应一种特定的合成语音，并为每条转录文本提供一个样本。数据集仅包含测试集，且所有样本均标记为“synthetic”，不包含任何人类语音。数据按语言组织，每种语言下包含一个 splits 目录，其中以 tar.gz 格式存储音频文件，文件名与原始 FLEURS 转录 ID 对应。

特点

该数据集涵盖八种语言：德语、英语、西班牙语、法语、意大利语、荷兰语、波兰语和瑞典语，体现多语言特性。其显著特点在于专注于合成语音检测的困难测试场景，所有样本均来自 TTS 系统，不含真实人声，这使其成为评估模型泛化能力的严苛基准。音频特征包括路径、数组和采样率（16000 Hz），标签统一为“synthetic”。此外，数据集还附带了录音元数据、转录文本和语音元数据文件，便于研究者深入分析。

使用方法

使用该数据集时，可通过 HuggingFace 的 load_dataset 函数加载特定语言的测试集，例如加载美式英语需指定配置为“en_us”。由于数据集采用自定义加载脚本，必须设置 trust_remote_code=True 参数。仅提供 test 分割，其他分割不存在。加载后的样本包含音频特征和标签字段，可直接用于评估合成语音检测模型的性能。研究者也可直接访问 data 目录下的原始文件，利用元数据文件进行自定义分析或扩展实验。

背景与挑战

背景概述

在语音合成技术迅猛发展的当下，深度伪造语音对信息安全构成了严峻挑战。FLEURS-HS VITS数据集由RealNetworks旗下KONTXT团队于2024年创建，作为《Synthetic speech detection with Wav2Vec 2.0 in various language settings》论文的核心组成部分，旨在推动多语言场景下合成语音检测的研究。该数据集以Google发布的FLEURS多语种语音语料库为基础，通过Google Cloud Text-To-Speech、Azure Text-To-Speech和Amazon Polly三大商用文本转语音引擎生成合成样本，覆盖德语、英语、西班牙语、法语、意大利语、荷兰语、波兰语和瑞典语共八种语言。其独特之处在于仅提供测试集，且每条转录文本对应一个VITS模型权重的合成音频，专门用于评估检测模型在未见过的合成语音上的泛化能力。该工作发表于ICASSP 2024 Workshop，为多语言反欺骗研究提供了标准化测试基准。

当前挑战

FLEURS-HS VITS数据集面临的挑战体现在两个层面。在领域问题层面，合成语音检测需要应对日益逼真的文本转语音技术——VITS模型能生成高度自然的语音，使得传统基于声学特征的检测方法失效；同时多语言场景要求模型具备跨语言泛化能力，避免因语言特定伪影导致检测偏差。在构建过程中，挑战主要源于数据来源的异质性：不同TTS服务商（如谷歌、微软、亚马逊）的合成语音在声学特性上存在差异，且VITS模型权重版本不一；此外，FLEURS原始语料涉及多语言、多口音和录音环境，需确保合成样本与真实样本在内容上严格对齐，避免因转录偏差引入混淆因素。数据集仅包含合成样本而无真实语音，这进一步考验检测模型在零样本场景下的鲁棒性。

常用场景

经典使用场景

作为多语言合成语音检测领域的一项关键资源，FLEURS-HS VITS 数据集专为评估语音伪造检测系统在跨语言场景下的泛化能力而设计。其经典使用场景聚焦于利用 Wav2Vec 2.0 等自监督预训练模型，在涵盖德语、英语、西班牙语、法语等八种语言的测试集上，对基于 VITS 架构的文本转语音系统生成的合成语音进行二分类判别。研究者通常将该数据集作为高难度测试基准，以衡量检测模型在面对未知合成算法时的鲁棒性。

衍生相关工作

基于该数据集，衍生出一系列具有深远影响的学术工作。其中最具代表性的是《Synthetic speech detection with Wav2Vec 2.0 in various language settings》一文，首次系统性地展示了自监督模型在跨语言合成语音检测中的潜力。此外，后续研究进一步探索了多任务学习、对比学习以及语言适配层等增强策略，有效提升了检测模型在低资源语言上的表现。这些工作共同构筑了多语言语音伪造检测的理论与方法论基石。

数据集最近研究