Common Crawl Question Answering (CCQA) dataset

CCQA数据集是由Meta AI和英属哥伦比亚大学合作创建的一个大规模开放领域问答数据集，基于Common Crawl项目，包含约1.3亿个多语言问答对，其中约6000万为英语数据点。该数据集旨在通过大规模、自然、多样且高质量的语料库，为问答任务预训练流行的语言模型。CCQA数据集通过schema.org的明确标注生成问答对，确保数据质量，并支持多种问答相关任务，如答案选择、评分和排名。此外，数据集还包含投票、多个竞争性答案、问题摘要和HTML标记等额外数据属性，适用于未来的多种研究应用。The CCQA dataset is a large-scale open-domain question answering dataset co-created by Meta AI and the University of British Columbia. Based on the Common Crawl project, it contains approximately 130 million multilingual question-answer pairs, among which about 60 million are English data points. The core goal of this dataset is to pre-train mainstream language models for question answering tasks by leveraging a large-scale, natural, diverse and high-quality corpus. The CCQA dataset generates question-answer pairs via explicit schema.org annotations to ensure data quality, and supports a wide range of question answering-related tasks such as answer selection, scoring and ranking. In addition, the dataset also includes additional data attributes such as votes, multiple competing answers, question summaries and HTML markup, which are applicable to a variety of future research applications.

搜集汇总

背景与挑战

背景概述

CCQA数据集是一个大规模开放领域问答数据集，由Meta AI和英属哥伦比亚大学基于Common Crawl项目合作创建，包含约1.3亿个多语言问答对（其中6000万为英语），旨在通过高质量、多样化的语料库支持问答模型的预训练。该数据集通过schema.org的明确标注生成问答对，确保数据质量，并支持多种问答相关任务，如答案选择、评分和排名，同时包含额外数据属性（如投票和问题摘要），适用于广泛的研究应用。