有没有相关的论文或文献参考？
这个数据集是基于什么背景创建的？
数据集的作者是谁？
能帮我联系到这个数据集的作者吗？
这个数据集如何下载？
Common Crawl|网络爬取数据集|文本挖掘数据集
收藏OpenDataLab2025-04-05 更新2024-05-09 收录
下载链接：
https://opendatalab.org.cn/OpenDataLab/Common_Crawl
下载链接
链接失效反馈资源简介：
Common Crawl 语料库包含在 12 年的网络爬取过程中收集的 PB 级数据。语料库包含原始网页数据、元数据提取和文本提取。 Common Crawl 数据存储在 Amazon Web Services 的公共数据集和全球多个学术云平台上。
提供机构：
OpenDataLab
创建时间：
2022-08-16
AI搜集汇总
数据集介绍
构建方式
Common Crawl数据集的构建基于大规模的网络爬虫技术，通过定期抓取互联网上的公开网页内容，形成一个庞大的文本数据库。该过程涉及复杂的爬虫算法和分布式计算系统，确保数据的广泛性和时效性。数据集的构建还包括对抓取内容的预处理，如去重、过滤和格式化，以提高数据的质量和可用性。
特点
Common Crawl数据集以其海量性和多样性著称，涵盖了全球范围内的网页内容，包括文本、图像和多媒体数据。其特点在于数据的实时更新和广泛覆盖，能够反映互联网内容的最新动态。此外，数据集的开放性和免费访问政策，使其成为研究者和开发者的重要资源，尤其在自然语言处理和数据挖掘领域具有广泛应用。
使用方法
Common Crawl数据集的使用方法多样，适用于多种研究和开发场景。研究者可以通过API接口或直接下载数据集进行分析，利用其丰富的文本数据进行语言模型训练、情感分析和信息检索等任务。开发者则可以利用数据集构建搜索引擎、推荐系统和内容管理系统。使用时需注意数据集的规模和复杂性，合理配置计算资源和数据处理策略，以确保高效和准确的结果。
背景与挑战
背景概述
Common Crawl数据集，自2011年由Gil Elbaz及其团队创建以来，已成为全球最大的公开可用网页存档之一。该数据集的核心研究问题在于如何高效地抓取、存储和分析海量的网页数据，以支持自然语言处理、机器学习和数据挖掘等领域的研究。Common Crawl每月从互联网上抓取数十亿网页，并将其以开放数据的形式提供给全球的研究者和开发者，极大地推动了相关领域的技术进步和创新。
当前挑战
尽管Common Crawl数据集在数据规模和开放性方面具有显著优势，但其构建和使用过程中仍面临诸多挑战。首先，数据的质量和一致性问题，由于抓取的网页内容多样且复杂，数据清洗和预处理成为一大难题。其次，数据存储和处理的效率问题，如何高效地存储和检索如此庞大的数据集，对计算资源和算法设计提出了高要求。此外，隐私和版权问题也是不可忽视的挑战，如何在保证数据开放性的同时，遵守相关法律法规，保护用户隐私和知识产权，是该数据集未来发展的重要课题。
发展历史
创建时间与更新
Common Crawl数据集创建于2011年，自那时起，它每年都会进行多次更新，以确保数据的时效性和广泛性。
重要里程碑
Common Crawl的首次发布标志着大规模公开网络数据集的诞生，为自然语言处理和机器学习领域提供了前所未有的资源。2012年，该数据集首次被用于构建大规模语言模型，显著推动了相关研究的发展。2015年，Common Crawl开始提供结构化数据，包括HTML、文本和元数据，进一步丰富了研究者的数据来源。
当前发展情况
当前，Common Crawl已成为全球最大的公开网络数据集之一，每月更新数十亿网页数据，涵盖多种语言和领域。它不仅支持了众多学术研究项目，还为工业界提供了宝贵的数据资源，推动了搜索引擎优化、内容分析和智能推荐系统的发展。Common Crawl的持续更新和扩展，使其在数据科学和人工智能领域中保持着重要的地位。
发展历程
- Common Crawl项目正式启动，旨在创建一个开放的网络数据集，供研究人员和开发者使用。
- 首次发布Common Crawl数据集，包含约50亿个网页的文本数据，标志着该项目进入实际应用阶段。
- Common Crawl数据集的规模显著扩大，涵盖了超过100亿个网页，成为全球最大的开放网络数据集之一。
- Common Crawl开始提供每月更新的数据集，确保数据的时效性和实用性，进一步推动了其在学术和工业界的应用。
- Common Crawl数据集被广泛应用于自然语言处理、机器学习和数据挖掘等领域，成为研究者和开发者的重要资源。
常用场景
经典使用场景
在自然语言处理领域，Common Crawl数据集被广泛用于大规模文本数据的预处理和分析。其经典使用场景包括但不限于语言模型训练、信息检索系统的构建以及跨语言翻译模型的开发。通过利用这一数据集，研究者能够获取海量的非结构化文本，从而为各种复杂的语言任务提供丰富的训练数据。
解决学术问题
Common Crawl数据集在解决学术研究问题方面具有显著意义。它为研究者提供了一个庞大的、多样化的文本资源库，有助于解决数据稀缺和数据偏差等常见问题。例如，通过使用Common Crawl，研究者可以训练出更加鲁棒和泛化能力强的语言模型，从而推动自然语言处理技术的发展。
衍生相关工作
基于Common Crawl数据集，许多经典工作得以展开。例如，研究者们开发了多种预训练语言模型，如BERT和GPT系列，这些模型在多个自然语言处理任务中表现出色。此外，Common Crawl还促进了跨领域研究，如数据挖掘和网络分析，推动了相关领域的技术进步和创新。
以上内容由AI搜集并总结生成
数据主题
5,000+
优质数据集
54 个
任务类型
进入经典数据集
热门数据集
Canadian Census
**Overview** The data package provides demographics for Canadian population groups according to multiple location categories: Forward Sortation Areas (FSAs), Census Metropolitan Areas (CMAs) and Census Agglomerations (CAs), Federal Electoral Districts (FEDs), Health Regions (HRs) and provinces. **Description** The data are available through the Canadian Census and the National Household Survey (NHS), separated or combined. The main demographic indicators provided for the population groups, stratified not only by location but also for the majority by demographical and socioeconomic characteristics, are population number, females and males, usual residents and private dwellings. The primary use of the data at the Health Region level is for health surveillance and population health research. Federal and provincial departments of health and human resources, social service agencies, and other types of government agencies use the information to monitor, plan, implement and evaluate programs to improve the health of Canadians and the efficiency of health services. Researchers from various fields use the information to conduct research to improve health. Non-profit health organizations and the media use the health region data to raise awareness about health, an issue of concern to all Canadians. The Census population counts for a particular geographic area representing the number of Canadians whose usual place of residence is in that area, regardless of where they happened to be on Census Day. Also included are any Canadians who were staying in that area on Census Day and who had no usual place of residence elsewhere in Canada, as well as those considered to be 'non-permanent residents'. National Household Survey (NHS) provides demographic data for various levels of geography, including provinces and territories, census metropolitan areas/census agglomerations, census divisions, census subdivisions, census tracts, federal electoral districts and health regions. In order to provide a comprehensive overview of an area, this product presents data from both the NHS and the Census. NHS data topics include immigration and ethnocultural diversity; aboriginal peoples; education and labor; mobility and migration; language of work; income and housing. 2011 Census data topics include population and dwelling counts; age and sex; families, households and marital status; structural type of dwelling and collectives; and language. The data are collected for private dwellings occupied by usual residents. A private dwelling is a dwelling in which a person or a group of persons permanently reside. Information for the National Household Survey does not include information for collective dwellings. Collective dwellings are dwellings used for commercial, institutional or communal purposes, such as a hotel, a hospital or a work camp. **Benefits** - Useful for canada public health stakeholders, for public health specialist or specialized public and other interested parties. for health surveillance and population health research. for monitoring, planning, implementation and evaluation of health-related programs. media agencies may use the health regions data to raise awareness about health, an issue of concern to all canadians. giving the addition of longitude and latitude in some of the datasets the data can be useful to transpose the values into geographical representations. the fields descriptions along with the dataset description are useful for the user to quickly understand the data and the dataset. **License Information** The use of John Snow Labs datasets is free for personal and research purposes. For commercial use please subscribe to the [Data Library](https://www.johnsnowlabs.com/marketplace/) on John Snow Labs website. The subscription will allow you to use all John Snow Labs datasets and data packages for commercial purposes. **Included Datasets** - [Canadian Population and Dwelling by FSA 2011](https://www.johnsnowlabs.com/marketplace/canadian-population-and-dwelling-by-fsa-2011) - This Canadian Census dataset covers data on population, total private dwellings and private dwellings occupied by usual residents by forward sortation area (FSA). It is enriched with the percentage of the population or dwellings versus the total amount as well as the geographical area, province, and latitude and longitude. The whole Canada's population is marked as 100, referring to 100% for the percentages. - [Detailed Canadian Population Statistics by CMAs and CAs 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-cmas-and-cas-2011) - This dataset covers the population statistics of Canada by Census Metropolitan Areas (CMAs) and Census Agglomerations (CAs). It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. - [Detailed Canadian Population Statistics by FED 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-fed-2011) - This dataset covers the population statistics of Canada from 2011 by Federal Electoral District of 2013 Representation Order. It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. - [Detailed Canadian Population Statistics by Health Region 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-health-region-2011) - This dataset covers the population statistics of Canada by health region. It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. - [Detailed Canadian Population Statistics by Province 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-province-2011) - This dataset covers the population statistics of Canada by provinces and territories. It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. **Data Engineering Overview** **We deliver high-quality data** - Each dataset goes through 3 levels of quality review - 2 Manual reviews are done by domain experts - Then, an automated set of 60+ validations enforces every datum matches metadata & defined constraints - Data is normalized into one unified type system - All dates, unites, codes, currencies look the same - All null values are normalized to the same value - All dataset and field names are SQL and Hive compliant - Data and Metadata - Data is available in both CSV and Apache Parquet format, optimized for high read performance on distributed Hadoop, Spark & MPP clusters - Metadata is provided in the open Frictionless Data standard, and its every field is normalized & validated - Data Updates - Data updates support replace-on-update: outdated foreign keys are deprecated, not deleted **Our data is curated and enriched by domain experts** Each dataset is manually curated by our team of doctors, pharmacists, public health & medical billing experts: - Field names, descriptions, and normalized values are chosen by people who actually understand their meaning - Healthcare & life science experts add categories, search keywords, descriptions and more to each dataset - Both manual and automated data enrichment supported for clinical codes, providers, drugs, and geo-locations - The data is always kept up to date – even when the source requires manual effort to get updates - Support for data subscribers is provided directly by the domain experts who curated the data sets - Every data source’s license is manually verified to allow for royalty-free commercial use and redistribution. **Need Help?** If you have questions about our products, contact us at [info@johnsnowlabs.com](mailto:info@johnsnowlabs.com).
Databricks 收录
HazyDet
HazyDet是由解放军工程大学等机构创建的一个大规模数据集，专门用于雾霾场景下的无人机视角物体检测。该数据集包含383,000个真实世界实例，收集自自然雾霾环境和正常场景中人工添加的雾霾效果，以模拟恶劣天气条件。数据集的创建过程结合了深度估计和大气散射模型，确保了数据的真实性和多样性。HazyDet主要应用于无人机在恶劣天气条件下的物体检测，旨在提高无人机在复杂环境中的感知能力。
arXiv 收录
LFW
人脸数据集;LFW数据集共有13233张人脸图像，每张图像均给出对应的人名，共有5749人，且绝大部分人仅有一张图片。每张图片的尺寸为250X250，绝大部分为彩色图像，但也存在少许黑白人脸图片。 URL: http://vis-www.cs.umass.edu/lfw/index.html#download
AI_Studio 收录
CE-CSL
CE-CSL数据集是由哈尔滨工程大学智能科学与工程学院创建的中文连续手语数据集，旨在解决现有数据集在复杂环境下的局限性。该数据集包含5,988个从日常生活场景中收集的连续手语视频片段，涵盖超过70种不同的复杂背景，确保了数据集的代表性和泛化能力。数据集的创建过程严格遵循实际应用导向，通过收集大量真实场景下的手语视频材料，覆盖了广泛的情境变化和环境复杂性。CE-CSL数据集主要应用于连续手语识别领域，旨在提高手语识别技术在复杂环境中的准确性和效率，促进聋人与听人社区之间的无障碍沟通。
arXiv 收录
MedDialog
MedDialog数据集（中文）包含了医生和患者之间的对话（中文）。它有110万个对话和400万个话语。数据还在不断增长，会有更多的对话加入。原始对话来自好大夫网。
github 收录