projecte-aina/parlament_parla_v3

--- license: cc-by-4.0 task_categories: - automatic-speech-recognition language: - ca tags: - central size_categories: - 10K<n<100K --- # Dataset Card for ParlamentParla v3 - Speech Corpus of Catalan Parliamentary Sessions A speech corpus composed of Catalan Parliamentary Sessions.The v3 and last version of the corpus includes both clean and other quality segments, divided into short segments (less than 30 seconds) and long segments (more than 30 seconds). The total dataset encompasses 1059h 48m 04s of speech, including 945h 51m 06s for the short segments and 113h 56m 58s for the long segments, with a total of 10.925.943 words. ## Table of Contents - [Dataset Details](#dataset-details) - [Dataset Description](#dataset-description) - [Dataset Sources](#dataset-sources) - [Uses](#uses) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits) - [Example Usage](#example-usage) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Data Collection and Processing](#data-collection-and-processing) - [Who are the Source Data Producers?](#source-data-producers) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Citation](#citation) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) ### Dataset Details ### Dataset Description This is the third version of the ParlamentParla speech corpus for Catalan: a collection of speech recordings with transcriptions intended for Automatic Speech Recognition (ASR) applications. In recent years, data in Catalan language has increased considerably. With this release we develop the third version of the ParlamentParla speech corpus, which will be very valuable mainly for training and evaluating speech recognition systems. We used the content of the Catalan Parlamentary sessions: the audio segments were extracted from recordings the Catalan Parliament ([Parlament de Catalunya](https://www.parlament.cat/)) plenary sessions. Taking advantage of the manual transcriptions, we created high quality audio segments in Catalan along with the aligned transcriptions. The extensive time span covered by the sessions, from January 23, 2008, to October 24, 2023, provides a broad range of linguistic phenomena and topics, further enriching the corpus. With 317 sessions in total, the corpus is substantial and should provide ample data for various research and development purposes in speech recognition. The final corpus has been extracted March 6, 2024. - **Curated by:** Language Technologies Unit at the Barcelona Supercomputing Center ([e-mail]) - **Funded by:** This work/research has been promoted and financed by the Government of Catalonia through the [Aina project](https://projecteaina.cat/). - **Shared by:** [More Information Needed] - **Language(s) (NLP):** ca-ce - **License:** [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/deed.es) ### Dataset Sources - **Repository:** - **Paper:** ### Uses The purpose of this dataset is mainly for training automatic speech recognition (ASR) models in Catalan. The language is limited to the plenary sessions of the parlament used to create the corpus and may not be representative to all domains. ## Dataset Structure ### Data Instances Each instance have the following structure: ```python DatasetDict({ clean_train: Dataset({ features: ['identifier','audio','segment_path','text'], num_rows: 164416 }) ``` Each data point is structured as: - Audio ID ```python >>data['clean_train_short'][0]['identifier'] 2753976_90753a8d81888d998484_405.96_411.15999999999997 ``` - Audio ```python >>data['clean_train_short'][0]['audio'] {'path': '/Users/sarahsolito/.cache/huggingface/datasets/downloads/extracted/9f760c175adf0af8127242f9468e48120f7682b20cf5c5813bfe481a108524bf/parlament_parla_v3/corpus/speech/2753976/2753976_90753a8d81888d998484_405.96_411.15999999999997.wav', 'array': array([-1.07421875e-02, -1.33972168e-02, -1.62353516e-02, ..., 1.64794922e-03, 3.05175781e-05, -4.02832031e-03]), 'sampling_rate': 16000} ``` - Relative Path ```python >>data['clean_train_short'][0]['segment_path'] data/parlament_parla_v3/output_segment/2753976/2753976_90753a8d81888d998484_405.96_411.15999999999997.wav ``` - Transcription ```python >>data['clean_train_short'][0]['text']) idò jo em tragaré el salmó oh uh no hi pensava ``` ### Data Fields - "identifier" : (string) &rarr; the unique audio identificator - "segment_path": (string) &rarr; the path to the audio - "audio": datasets.Audio(sampling_rate=16000) &rarr; the decoded audio array, and the sampling rate. - "text": (string) &rarr; clean version of the transcription ### Data Splits The dataset consists of a train, dev and test splits. The stat details are as follows: | Subcorpus | Duration | |------------------ |-----------| | other_test_short | 13:42:44 | | other_dev_short | 13:13:45 | | other_train_short | 507:27:34 | |*other total_short*| 534:24:03 | | clean_test_short | 10:44:19 | | clean_dev_short | 10:23:30 | | clean_train_short | 390:19:12 | |*clean total_short*| 411:27:03 | |*Total* | 945:51:06 | | Subcorpus | Duration | |-------------------|-----------| | other_test_long | 01:41:29 | | other_dev_long | 01:51:30 | | other_train_long | 72:35:10 | |*other total_long* | 76:08:10 | | clean_test_long | 00:50:15 | | clean_dev_long | 00:46:44 | | clean_train_long | 36:11:46 | |*clean total_long* | 37:48:47 | |*Total* | 113:56:58 | :04 ### Example Usage To load a specific split ,for example, the training split do: ```python from datasets import load_dataset data = load_dataset("projecte-aina/parlament_parla_v3",split="clean_train_short") ``` ## Dataset Creation ### Curation Rationale The directory called "speech" contains all the speech files of the corpus. The files in the speech directory are divided into the "clean" and the "other" directories. ### Source Data The content belongs to the Catalan Parliament and the data is released conforming their [terms of use](https://www.parlament.cat/pcat/serveis-parlament/avis-legal/). ### Data Collection and Processing The dataset's transcriptions are released in a clean version. The clean versions have been normalized at an orthographic level in lower-case. The normalization process was performed removing punctuation marks and characters that are not present in the Catalan alphabet. Number expansion was also perfomed. In order to obtain a corpus of the highest possible quality, we also apply automatic language detection processes to each segment to prevent code-switching, and evaluate the quality of the transcriptions to eliminate both low quality segments and those that are not in Catalan. ### Who are the source data producers? The content belongs to the Catalan Parliament and the data is released conforming their [terms of use](https://www.parlament.cat/pcat/serveis-parlament/avis-legal/). ### Annotations The dataset doesn't contain any additional annotation. ### Personal and Sensitive Information The dataset consists of Catalan parliamentary speeches and their transcription. The dataset contains no personal information except for speech, which is considered personal data. Consequently, the speakers' voices in this corpus have been subjected to anonymization treatment in compliance with applicable regulations, such as the General Data Protection Regulation (GDPR) in the European Union. You agree to not attempt to determine the identity of speakers in this dataset. ### Citation ``` @misc{bscib32024, title={ParlamentParla v3 - Speech Corpus of Catalan Parliamentary Sessions}, author={Baybars, Kulebi}, publisher={Barcelona Supercomputing Center}, year={2024}, url={}, } ``` ## Considerations for Using the Data ### Social Impact of Dataset ParlamentParla_v3 is a source of speech data that will be valuable in development of speech technologies for Catalan language and its varieties. ### Discussion of Biases No specific bias mitigation strategies were applied to this dataset. Inherent biases may exist within the data. ### Other Known Limitations Speakers, their gender and age are not identified and one or more speakers could be speaking in the same recording. For these reasons, we don't know the total number of speakers in the corpus and their gender/age.license: CC-BY-4.0 task_categories: - 自动语音识别（Automatic Speech Recognition, ASR） language: - 加泰罗尼亚语（ca） tags: - 中央 size_categories: - 10K<n<100K # ParlamentParla v3 数据集卡片：加泰罗尼亚议会会议语音语料库 本语料库由加泰罗尼亚议会会议语音数据构成，其v3版本亦是当前最新版本，包含高质量（clean）与普通质量的语音片段，并按时长划分为短片段（时长小于30秒）与长片段（时长大于30秒）。本数据集总语音时长为1059小时48分04秒，其中短片段总时长945小时51分06秒，长片段总时长113小时56分58秒，总词汇量达10,925,943词。 ## 目录 - [数据集详情](#数据集详情) - [数据集描述](#数据集描述) - [数据集来源](#数据集来源) - [使用场景](#使用场景) - [数据集结构](#数据集结构) - [数据实例](#数据实例) - [数据字段](#数据字段) - [数据划分](#数据划分) - [示例用法](#示例用法) - [数据集构建](#数据集构建) - [整理逻辑](#整理逻辑) - [源数据](#源数据) - [数据收集与处理](#数据收集与处理) - [源数据生产者是谁？](#源数据生产者是谁？) - [标注信息](#标注信息) - [个人与敏感信息](#个人与敏感信息) - [引用格式](#引用格式) - [数据集使用注意事项](#数据集使用注意事项) - [数据集的社会影响](#数据集的社会影响) - [偏倚讨论](#偏倚讨论) - [已知其他局限性](#已知其他局限性) ### 数据集详情 ### 数据集描述 本语料库为加泰罗尼亚语ParlamentParla语音语料库的第三个版本，是面向自动语音识别（Automatic Speech Recognition, ASR）应用场景构建的带转录文本的语音录音合集。 近年来，加泰罗尼亚语语音数据资源增长显著。本次发布的ParlamentParla v3语料库可广泛应用于语音识别系统的训练与评估，具备极高的科研与应用价值。 本数据集取材自加泰罗尼亚议会会议内容：语音片段提取自加泰罗尼亚议会（[Parlament de Catalunya](https://www.parlament.cat/)）全体会议的公开录音。依托人工转录文本，我们生成了与语音对齐的高质量加泰罗尼亚语语音片段。 本次数据集覆盖的会议时间跨度为2008年1月23日至2023年10月24日，涵盖了丰富的语言现象与讨论主题，进一步丰富了语料库的多样性。本数据集共包含317场会议，规模充足，可为语音识别领域的各类研发与研究工作提供足量数据支持。本语料库最终于2024年3月6日完成提取。 - **整理方：** 巴塞罗那超级计算中心（Barcelona Supercomputing Center, BSC）语言技术团队（邮箱：[e-mail]） - **资助方：** 本研究由加泰罗尼亚政府通过[Aina项目](https://projecteaina.cat/)发起并资助。 - **共享方：** [需补充更多信息] - **自然语言处理所用语言：** ca-ce - **许可证：** [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/deed.es) ### 数据集来源 - **代码仓库：** - **相关论文：** ### 使用场景 本数据集主要用于训练加泰罗尼亚语自动语音识别（ASR）模型。由于数据仅取材自议会全体会议，其语言风格与主题可能无法覆盖所有应用领域。 ## 数据集结构 ### 数据实例 每条数据实例的结构如下： python DatasetDict({ clean_train: Dataset({ features: ['identifier','audio','segment_path','text'], num_rows: 164416 }) 每个数据点的结构如下： - 音频唯一标识符 python >>data['clean_train_short'][0]['identifier'] 2753976_90753a8d81888d998484_405.96_411.15999999999997 - 音频数据 python >>data['clean_train_short'][0]['audio'] {'path': '/Users/sarahsolito/.cache/huggingface/datasets/downloads/extracted/9f760c175adf0af8127242f9468e48120f7682b20cf5c5813bfe481a108524bf/parlament_parla_v3/corpus/speech/2753976/2753976_90753a8d81888d998484_405.96_411.15999999999997.wav', 'array': array([-1.07421875e-02, -1.33972168e-02, -1.62353516e-02, ..., 1.64794922e-03, 3.05175781e-05, -4.02832031e-03]), 'sampling_rate': 16000} - 相对存储路径 python >>data['clean_train_short'][0]['segment_path'] data/parlament_parla_v3/output_segment/2753976/2753976_90753a8d81888d998484_405.96_411.15999999999997.wav - 转录文本 python >>data['clean_train_short'][0]['text']) idò jo em tragaré el salmó oh uh no hi pensava ### 数据字段 - "identifier" : (字符串类型) → 音频唯一标识符 - "segment_path": (字符串类型) → 音频文件存储路径 - "audio": datasets.Audio(sampling_rate=16000) → 解码后的音频数组与采样率 - "text": (字符串类型) → 标准化后的转录文本 ### 数据划分 本数据集包含训练集、开发集与测试集三个划分，详细统计信息如下： | 子语料库 | 时长 | |-------------------------|---------------| | other_test_short | 13小时42分44秒 | | other_dev_short | 13小时13分45秒 | | other_train_short | 507小时27分34秒| | *other total_short* | 534小时24分03秒| | clean_test_short | 10小时44分19秒 | | clean_dev_short | 10小时23分30秒 | | clean_train_short | 390小时19分12秒| | *clean total_short* | 411小时27分03秒| | *总时长（短片段）* | 945小时51分06秒| | 子语料库 | 时长 | |-------------------------|---------------| | other_test_long | 1小时41分29秒 | | other_dev_long | 1小时51分30秒 | | other_train_long | 72小时35分10秒 | | *other total_long* | 76小时08分10秒 | | clean_test_long | 0小时50分15秒 | | clean_dev_long | 0小时46分44秒 | | clean_train_long | 36小时11分46秒 | | *clean total_long* | 37小时48分47秒 | | *总时长（长片段）* | 113小时56分58秒| :04 ### 示例用法 若需加载指定数据划分（例如清洁类短片段训练集），可使用如下代码： python from datasets import load_dataset data = load_dataset("projecte-aina/parlament_parla_v3",split="clean_train_short") ## 数据集构建 ### 整理逻辑 语料库的“speech”目录包含所有语音文件，该目录下的文件进一步划分为“clean”（高质量）与“other”（普通质量）两个子目录。 ### 源数据 本数据集内容归属于加泰罗尼亚议会，发布时已遵循其[使用条款](https://www.parlament.cat/pcat/serveis-parlament/avis-legal/)。 ### 数据收集与处理 本数据集的转录文本已发布为标准化版本。标准化过程包括：将文本转换为小写的正字法规范、移除标点符号与加泰罗尼亚语字母以外的字符、对数字进行展开处理。 为确保语料库质量，我们还对每个片段进行自动语言检测，以避免语码转换内容，并评估转录文本质量，剔除低质量片段与非加泰罗尼亚语内容。 ### 源数据生产者是谁？ 本数据集的内容归属于加泰罗尼亚议会，发布时已遵循其[使用条款](https://www.parlament.cat/pcat/serveis-parlament/avis-legal/)。 ### 标注信息 本数据集未包含额外标注内容。 ### 个人与敏感信息 本数据集包含加泰罗尼亚议会演讲语音及其转录文本。除语音本身属于个人数据外，数据集未包含其他个人信息。因此，语料库中的演讲者语音已按照欧盟《通用数据保护条例》（General Data Protection Regulation, GDPR）等适用法规进行了匿名化处理。使用者承诺不会尝试识别本数据集中的演讲者身份。 ### 引用格式 @misc{bscib32024, title={ParlamentParla v3 - Speech Corpus of Catalan Parliamentary Sessions}, author={Baybars, Kulebi}, publisher={Barcelona Supercomputing Center}, year={2024}, url={}, } ## 数据集使用注意事项 ### 数据集的社会影响 ParlamentParla_v3是加泰罗尼亚语及其变体语音技术研发的重要语音数据资源。 ### 偏倚讨论 本数据集未采用特定的偏倚缓解策略，数据中可能存在固有偏倚。 ### 已知其他局限性 本数据集未对演讲者的身份、性别与年龄进行标注，且单条录音中可能包含多位演讲者。因此，无法获知语料库中演讲者的总人数及其性别、年龄分布。