community-datasets/youtube_caption_corrections

--- annotations_creators: - expert-generated - machine-generated language_creators: - machine-generated language: - en license: - mit multilinguality: - monolingual size_categories: - 10K<n<100K source_datasets: - original task_categories: - other - text-generation - fill-mask task_ids: - slot-filling pretty_name: YouTube Caption Corrections tags: - token-classification-of-text-errors dataset_info: features: - name: video_ids dtype: string - name: default_seq sequence: string - name: correction_seq sequence: string - name: diff_type sequence: class_label: names: '0': NO_DIFF '1': CASE_DIFF '2': PUNCUATION_DIFF '3': CASE_AND_PUNCUATION_DIFF '4': STEM_BASED_DIFF '5': DIGIT_DIFF '6': INTRAWORD_PUNC_DIFF '7': UNKNOWN_TYPE_DIFF '8': RESERVED_DIFF splits: - name: train num_bytes: 355978891 num_examples: 10769 download_size: 49050406 dataset_size: 355978891 configs: - config_name: default data_files: - split: train path: data/train-* --- # Dataset Card for YouTube Caption Corrections ## Table of Contents - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Dataset Description - **Homepage:** https://github.com/2dot71mily/youtube_captions_corrections - **Repository:** https://github.com/2dot71mily/youtube_captions_corrections - **Paper:** [N/A] - **Leaderboard:** [N/A] - **Point of Contact:** Emily McMilin ### Dataset Summary This dataset is built from pairs of YouTube captions where both an auto-generated and a manually-corrected caption are available for a single specified language. It currently only in English, but scripts at repo support other languages. The motivation for creating it was from viewing errors in auto-generated captions at a recent virtual conference, with the hope that there could be some way to help correct those errors. The dataset in the repo at https://github.com/2dot71mily/youtube_captions_corrections records in a non-destructive manner all the differences between an auto-generated and a manually-corrected caption for thousands of videos. The dataset here focuses on the subset of those differences which are mutual and have the same size in token length difference, which means it excludes token insertion or deletion differences between the two captions. Therefore dataset here remains a non-destructive representation of the original auto-generated captions, but excludes some of the differences that are found in the manually-corrected captions. ### Supported Tasks and Leaderboards - `token-classification`: The tokens in `default_seq` are from the auto-generated YouTube captions. If `diff_type` is labeled greater than `0` at a given index, then the associated token in same index in the `default_seq` was found to be different to the token in the manually-corrected YouTube caption, and therefore we assume it is an error. A model can be trained to learn when there are errors in the auto-generated captions. - `slot-filling`: The `correction_seq` is sparsely populated with tokens from the manually-corrected YouTube captions in the locations where there was found to be a difference to the token in the auto-generated YouTube captions. These 'incorrect' tokens in the `default_seq` can be masked in the locations where `diff_type` is labeled greater than `0`, so that a model can be trained to hopefully find a better word to fill in, rather than the 'incorrect' one. End to end, the models could maybe first identify and then replace (with suitable alternatives) errors in YouTube and other auto-generated captions that are lacking manual corrections ### Languages English ## Dataset Structure ### Data Instances If `diff_type` is labeled greater than `0` at a given index, then the associated token in same index in the `default_seq` was found to have a difference to the token in the manually-corrected YouTube caption. The `correction_seq` is sparsely populated with tokens from the manually-corrected YouTube captions at those locations of differences. `diff_type` labels for tokens are as follows: 0: No difference 1: Case based difference, e.g. `hello` vs `Hello` 2: Punctuation difference, e.g. `hello` vs `hello` 3: Case and punctuation difference, e.g. `hello` vs `Hello,` 4: Word difference with same stem, e.g. `thank` vs `thanked` 5: Digit difference, e.g. `2` vs `two` 6: Intra-word punctuation difference, e.g. `autogenerated` vs `auto-generated` 7: Unknown type of difference, e.g. `laughter` vs `draft` 8: Reserved for unspecified difference { 'video_titles': '_QUEXsHfsA0', 'default_seq': ['you', 'see', "it's", 'a', 'laughter', 'but', 'by', 'the', 'time', 'you', 'see', 'this', 'it', "won't", 'be', 'so', 'we', 'have', 'a', 'big'] 'correction_seq': ['', 'see,', '', '', 'draft,', '', '', '', '', '', 'read', 'this,', '', '', 'be.', 'So', '', '', '', ''] 'diff_type': [0, 2, 0, 0, 7, 0, 0, 0, 0, 0, 7, 2, 0, 0, 2, 1, 0, 0, 0, 0] } ### Data Fields - 'video_ids': Unique ID used by YouTube for each video. Can paste into `https://www.youtube.com/watch?v=<{video_ids}` to see video - 'default_seq': Tokenized auto-generated YouTube captions for the video - 'correction_seq': Tokenized manually-corrected YouTube captions only at those locations, where there is a difference between the auto-generated and manually-corrected captions - 'diff_type': A value greater than `0` at every token where there is a difference between the auto-generated and manually-corrected captions ### Data Splits No data splits ## Dataset Creation ### Curation Rationale It was created after viewing errors in auto-generated captions at a recent virtual conference, with the hope that there could be some way to help correct those errors. ### Source Data #### Initial Data Collection and Normalization All captions are requested via `googleapiclient` and `youtube_transcript_api` at the `channel_id` and language granularity, using scripts written at https://github.com/2dot71mily/youtube_captions_corrections. The captions are tokenized on spaces and the manually-corrected sequence has here been reduced to only include differences between it and the auto-generated sequence. #### Who are the source language producers? Auto-generated scripts are from YouTube and the manually-corrected scripts are from creators, and any support they may have (e.g. community or software support) ### Annotations #### Annotation process Scripts at repo, https://github.com/2dot71mily/youtube_captions_corrections take a diff of the two captions and use this to create annotations. #### Who are the annotators? YouTube creators, and any support they may have (e.g. community or software support) ### Personal and Sensitive Information All content publicly available on YouTube ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators Emily McMilin ### Licensing Information MIT License ### Citation Information https://github.com/2dot71mily/youtube_captions_corrections ### Contributions Thanks to [@2dot71mily](https://github.com/2dot71mily) for adding this dataset.该数据集名为YouTube Caption Corrections，包含自动生成的YouTube字幕与人工校正后的字幕对，主要用于英语文本生成和填充掩码等任务。数据集通过video_ids、default_seq、correction_seq和diff_type等字段记录了两种字幕之间的差异。此数据集由Emily McMilin策划，遵循MIT许可证。

原始信息汇总

数据集概述

数据集摘要

该数据集包含YouTube视频的字幕修正信息，包括自动生成的字幕和手动修正的字幕。数据集目前仅支持英语，但脚本支持其他语言。数据集的创建动机是希望找到一种方法来修正自动生成字幕中的错误。

支持的任务和排行榜

• token-classification：识别自动生成字幕中的错误。
• slot-filling：在自动生成字幕中找到错误并替换为更合适的词。

语言

英语

数据集结构

数据实例

数据实例包含视频ID、自动生成字幕序列、手动修正字幕序列和差异类型。差异类型标签如下：

• 0: 无差异
• 1: 大小写差异
• 2: 标点差异
• 3: 大小写和标点差异
• 4: 词干相同但词形不同
• 5: 数字差异
• 6: 词内标点差异
• 7: 未知类型差异
• 8: 保留未指定差异

数据字段

• video_ids：YouTube视频的唯一ID。
• default_seq：视频的自动生成字幕序列。
• correction_seq：手动修正字幕序列，仅在自动生成字幕和手动修正字幕存在差异的位置包含内容。
• diff_type：标记自动生成字幕和手动修正字幕之间差异的类型。

数据分割

数据集未进行分割。

数据集创建

数据收集和规范化

字幕通过googleapiclient和youtube_transcript_api从YouTube获取，并使用脚本进行处理。

数据来源

自动生成字幕来自YouTube，手动修正字幕来自视频创作者及其支持团队。

标注过程

使用脚本对自动生成字幕和手动修正字幕进行差异比较，并生成标注。

标注者

视频创作者及其支持团队。

使用数据的注意事项

数据集的社会影响

[更多信息待补充]

数据集的偏见讨论

[更多信息待补充]

其他已知限制

[更多信息待补充]

附加信息

数据集创建者

Emily McMilin

许可信息

MIT许可证

引用信息

数据集GitHub仓库链接

贡献

感谢@2dot71mily添加此数据集。

搜集汇总

数据集介绍

构建方式

该数据集源自YouTube平台上同时具备自动生成字幕与人工校正字幕的视频资源，目前仅涵盖英语语料。构建过程中，研究者通过调用googleapiclient与youtube_transcript_api接口，以频道ID和语言为粒度获取原始字幕数据。随后，对自动生成字幕与人工校正字幕进行基于空格的令牌化处理，并利用差异比对算法（diff）提取两者之间的非破坏性差异。为确保数据结构的对称性，数据集仅保留那些令牌长度一致的差异对，排除了令牌插入或删除的情况。最终，每个视频实例包含视频ID、自动生成令牌序列、仅记录差异位置的人工校正令牌序列，以及细粒度的差异类型标签，涵盖大小写、标点、词干、数字等八类差异。

特点

该数据集的核心特色在于其精细化的差异标注体系，通过八类差异标签（如大小写差异、标点差异、词干差异等）对自动生成字幕中的错误进行详尽分类，为令牌级别的错误识别与纠正提供了高粒度监督信号。数据集保留了原始自动生成字幕的完整结构，仅在差异位置稀疏填充人工校正令牌，这种非破坏性设计使得模型既能学习错误定位，又能探索语义替换。此外，所有数据均来源于YouTube公开视频，具备真实场景下的噪声分布与语言多样性，尤其适用于构建端到端的字幕纠错系统，涵盖错误检测与智能填充两大任务。

使用方法

该数据集主要支持令牌分类与槽位填充两类任务。在令牌分类任务中，可将自动生成令牌序列（default_seq）作为输入，差异标签（diff_type）作为训练目标，训练模型识别字幕中的错误位置。在槽位填充任务中，可对差异位置（diff_type>0）的令牌进行掩码处理，利用人工校正令牌（correction_seq）作为监督，训练模型生成更优的替换词。实践中，研究者可先使用令牌分类模型定位错误，再通过掩码语言模型进行纠正，最终实现自动生成字幕的智能化修复。数据集以HuggingFace标准格式提供，支持直接加载与批次处理，便于集成到现有深度学习框架中。

背景与挑战

背景概述

在多媒体内容爆炸式增长的时代，自动生成字幕技术虽大幅提升了视频的可访问性，但其在语音识别、语境理解等方面的固有缺陷常导致文本错误，进而影响听障人士及非母语用户的观看体验。YouTube Caption Corrections数据集由Emily McMilin于近年创建，旨在系统性地记录和分类YouTube平台中自动生成字幕与人工修正字幕之间的差异。该数据集聚焦于英文视频，通过非破坏性方式保留原始自动生成字幕的序列结构，并标注了大小写、标点、词干、数字等八类差异类型，为字幕纠错研究提供了精细化的标注资源。其核心研究问题在于如何利用人工修正的监督信号训练模型，以自动识别并纠正自动生成字幕中的错误，从而提升语音识别后处理与文本生成领域的鲁棒性。该数据集的出现为低资源场景下的字幕质量优化开辟了新路径，并推动了错误检测与序列填充等自然语言处理任务的发展。

当前挑战

当前该数据集面临的核心挑战体现在两个层面。在领域问题层面，自动生成字幕的错误类型高度异构，包括同音词混淆、语法错误、专有名词误识等，而现有标注仅覆盖八类差异，难以穷举所有错误模式，限制了模型对复杂语义错误的泛化能力。此外，数据集仅保留等长序列中的差异，排除了插入或删除型错误，这使得模型无法学习到字幕中词序调整或内容增删的修正逻辑。在构建过程层面，数据来源依赖于YouTube创作者的手动修正，其修正标准不一、质量参差，且仅涵盖英语语料，缺乏多语言与跨领域视频的多样性。同时，人工标注的稀疏性（仅标注差异位置）使得模型在无差异区域缺乏负样本监督，易导致误检或漏检。这些因素共同制约了数据集在真实场景中端到端字幕纠错任务的实用性与可迁移性。

常用场景

经典使用场景

该数据集的核心应用场景在于训练能够识别并分类自动生成字幕中各类错误的序列标注模型。通过将自动字幕与人工修正字幕进行对齐，数据集为每个词元标注了差异类型，包括大小写、标点、词干变化、数字替换等八种错误类别。研究者可借此构建端到端的字幕错误检测与纠正流水线，首先利用token分类任务定位错误词元，再通过slot-filling任务生成合适的修正词。这一范式为提升语音识别后处理、视频字幕自动校对等任务提供了高质量的监督信号。

解决学术问题

该数据集直面自动语音识别（ASR）系统在真实场景中产生的字幕错误问题，弥补了现有语料库多聚焦于干净文本或人工标注错误类型的空白。它系统性地量化了自动字幕与人工修正之间的差异模式，使研究者能够深入分析ASR错误的分布规律与语言学特征。通过提供细粒度的错误分类标签，该数据集推动了从简单错误检测向错误类型识别与针对性修正的学术演进，为构建更鲁棒的字幕生成与后处理模型奠定了实证基础。

衍生相关工作

该数据集衍生了多个方向的研究工作。在模型层面，基于其token分类与slot-filling双任务设计，催生了结合预训练语言模型（如BERT、T5）进行上下文感知错误纠正的系列方法。在数据层面，其差异类型分类体系启发了面向多语言ASR错误分析的数据集构建范式，推动了对非英语字幕错误模式的跨语言研究。此外，该数据集的构建脚本与管道设计已被复用于其他视频平台的字幕质量评估任务，形成了从数据采集、差异对齐到模型训练的完整开源工具链。