有没有相关的论文或文献参考？
这个数据集是基于什么背景创建的？
数据集的作者是谁？
能帮我联系到这个数据集的作者吗？
这个数据集如何下载？
OSCAR|自然语言处理数据集|机器学习数据集
收藏OpenDataLab2025-04-05 更新2024-05-09 收录
下载链接：
https://opendatalab.org.cn/OpenDataLab/OSCAR
下载链接
链接失效反馈资源简介：
OSCAR 或 Open Super-large Crawled ALMAnaCH 语料库是使用 goclassy 架构对 Common Crawl 语料库进行语言分类和过滤得到的一个庞大的多语言语料库。用于训练 BART 等多语言模型的数据集包含 138 GB 的文本。
提供机构：
OpenDataLab
创建时间：
2022-08-10
AI搜集汇总
数据集介绍
构建方式
OSCAR数据集的构建基于大规模的网络文本资源，通过自动化的数据清洗和筛选过程，确保了数据的高质量和多样性。该数据集采用了先进的自然语言处理技术，对原始文本进行了分词、去重、过滤等处理，从而生成了一系列结构化的文本数据。此外，OSCAR还结合了多语言处理能力，涵盖了多种语言的文本数据，为跨语言研究提供了丰富的资源。
特点
OSCAR数据集以其大规模、多语言和高多样性著称。该数据集包含了来自全球各地的多种语言文本，涵盖了广泛的主题和领域，为语言学、机器学习和数据科学研究提供了宝贵的资源。其数据质量经过严格筛选和处理，确保了研究结果的可靠性和准确性。此外，OSCAR的开放性和易用性也使其成为学术界和工业界广泛使用的数据集之一。
使用方法
OSCAR数据集适用于多种自然语言处理任务，包括但不限于文本分类、情感分析、机器翻译和语言模型训练。研究人员可以通过访问OSCAR的官方网站或相关数据平台，下载所需的数据子集进行实验和分析。在使用过程中，建议根据具体研究需求选择合适的语言和文本类型，并结合相应的预处理工具进行数据清洗和格式化，以确保实验结果的有效性和可重复性。
背景与挑战
背景概述
OSCAR数据集，由OpenAI于2020年推出，旨在为自然语言处理领域提供一个大规模、高质量的预训练语言模型。该数据集的核心研究问题是如何在海量文本数据中提取有用的语言特征，以提升模型在多种任务上的表现。OSCAR的创建标志着自然语言处理技术从依赖特定任务数据集向通用预训练模型的转变，极大地推动了诸如文本分类、机器翻译和问答系统等应用的发展。
当前挑战
OSCAR数据集在构建过程中面临诸多挑战。首先，数据清洗和去重是确保数据质量的关键步骤，但处理数十亿条文本数据的技术复杂性极高。其次，如何在保持数据多样性的同时，避免引入偏见和错误信息，是该数据集面临的另一大挑战。此外，OSCAR的广泛应用也带来了模型解释性和公平性问题，如何在不同文化和语言背景下保持模型的公正性和透明度，是当前研究的重要方向。
发展历史
创建时间与更新
OSCAR数据集由Hugging Face于2020年首次发布，旨在为自然语言处理领域提供一个大规模、多语言的文本数据集。该数据集自发布以来，经历了多次更新，以适应不断变化的研究需求和技术进步。
重要里程碑
OSCAR数据集的一个重要里程碑是其首次发布，这一事件标志着多语言文本数据处理进入了一个新的阶段。随后，OSCAR数据集的持续更新和扩展，特别是在2021年引入的版本中，增加了对更多语言的支持和数据清洗技术的改进，进一步提升了其在自然语言处理研究中的应用价值。此外，OSCAR数据集在2022年与多个国际研究项目合作，推动了跨语言模型的训练和评估，成为多语言自然语言处理领域的重要资源。
当前发展情况
当前，OSCAR数据集已成为自然语言处理领域中不可或缺的资源，广泛应用于语言模型训练、文本分类、机器翻译等多个研究方向。其多语言特性和高质量的数据清洗技术，使得OSCAR数据集在推动全球语言多样性研究和跨文化交流方面发挥了重要作用。随着技术的不断进步，OSCAR数据集预计将继续扩展其语言覆盖范围和数据质量，为未来的自然语言处理研究提供更加丰富和可靠的数据支持。
发展历程
- OSCAR数据集首次发表，由Hugging Face团队推出，旨在为自然语言处理任务提供大规模的文本数据。
- OSCAR数据集首次应用于多语言预训练模型，显著提升了模型在跨语言任务中的表现。
- OSCAR数据集的版本更新，增加了更多语言的支持，并优化了数据质量，进一步推动了多语言NLP研究的发展。
常用场景
经典使用场景
在自然语言处理领域，OSCAR数据集以其庞大的多语言文本资源而著称。该数据集广泛应用于语言模型训练、文本分类和信息检索等经典场景。通过OSCAR，研究者能够构建和优化多语言模型，提升跨语言理解和生成的能力。其丰富的语料库为机器翻译、情感分析和文本生成等任务提供了坚实的基础。
实际应用
在实际应用中，OSCAR数据集被广泛用于构建多语言搜索引擎、智能客服系统和跨语言内容推荐系统。例如，通过OSCAR训练的模型可以实现多语言文本的自动分类和情感分析，帮助企业更好地理解全球用户的反馈和需求。此外，OSCAR还支持多语言机器翻译系统的开发，提升了跨语言沟通的效率和准确性。
衍生相关工作
基于OSCAR数据集，研究者们开发了多种多语言预训练模型，如mBERT和XLM-R，这些模型在多个自然语言处理任务中表现优异。此外，OSCAR还激发了多语言数据集的标准化和共享机制的研究，推动了多语言资源的开放获取和公平使用。这些衍生工作不仅丰富了自然语言处理的工具库，也为全球语言技术的均衡发展提供了有力支持。
以上内容由AI搜集并总结生成
数据主题
5,000+
优质数据集
54 个
任务类型
进入经典数据集
热门数据集
Materials Project 在线材料数据库
Materials Project 是一个由伯克利加州大学和劳伦斯伯克利国家实验室于 2011 年共同发起的大型开放式在线材料数据库。这个项目的目标是利用高通量第一性原理计算，为超过百万种无机材料提供全面的性能数据、结构信息和计算模拟结果，以此加速新材料的发现和创新过程。数据库中的数据不仅包括晶体结构和能量特性，还涵盖了电子结构和热力学性质等详尽信息，为研究人员提供了丰富的材料数据资源。相关论文成果为「Commentary: The Materials Project: A materials genome approach to accelerating materials innovation」。
超神经 收录
OMIM (Online Mendelian Inheritance in Man)
OMIM是一个包含人类基因和遗传疾病信息的在线数据库。它提供了详细的遗传疾病描述、基因定位、相关文献和临床信息。数据集内容包括疾病名称、基因名称、基因定位、遗传模式、临床特征、相关文献引用等。
www.omim.org 收录
Canadian Census
**Overview** The data package provides demographics for Canadian population groups according to multiple location categories: Forward Sortation Areas (FSAs), Census Metropolitan Areas (CMAs) and Census Agglomerations (CAs), Federal Electoral Districts (FEDs), Health Regions (HRs) and provinces. **Description** The data are available through the Canadian Census and the National Household Survey (NHS), separated or combined. The main demographic indicators provided for the population groups, stratified not only by location but also for the majority by demographical and socioeconomic characteristics, are population number, females and males, usual residents and private dwellings. The primary use of the data at the Health Region level is for health surveillance and population health research. Federal and provincial departments of health and human resources, social service agencies, and other types of government agencies use the information to monitor, plan, implement and evaluate programs to improve the health of Canadians and the efficiency of health services. Researchers from various fields use the information to conduct research to improve health. Non-profit health organizations and the media use the health region data to raise awareness about health, an issue of concern to all Canadians. The Census population counts for a particular geographic area representing the number of Canadians whose usual place of residence is in that area, regardless of where they happened to be on Census Day. Also included are any Canadians who were staying in that area on Census Day and who had no usual place of residence elsewhere in Canada, as well as those considered to be 'non-permanent residents'. National Household Survey (NHS) provides demographic data for various levels of geography, including provinces and territories, census metropolitan areas/census agglomerations, census divisions, census subdivisions, census tracts, federal electoral districts and health regions. In order to provide a comprehensive overview of an area, this product presents data from both the NHS and the Census. NHS data topics include immigration and ethnocultural diversity; aboriginal peoples; education and labor; mobility and migration; language of work; income and housing. 2011 Census data topics include population and dwelling counts; age and sex; families, households and marital status; structural type of dwelling and collectives; and language. The data are collected for private dwellings occupied by usual residents. A private dwelling is a dwelling in which a person or a group of persons permanently reside. Information for the National Household Survey does not include information for collective dwellings. Collective dwellings are dwellings used for commercial, institutional or communal purposes, such as a hotel, a hospital or a work camp. **Benefits** - Useful for canada public health stakeholders, for public health specialist or specialized public and other interested parties. for health surveillance and population health research. for monitoring, planning, implementation and evaluation of health-related programs. media agencies may use the health regions data to raise awareness about health, an issue of concern to all canadians. giving the addition of longitude and latitude in some of the datasets the data can be useful to transpose the values into geographical representations. the fields descriptions along with the dataset description are useful for the user to quickly understand the data and the dataset. **License Information** The use of John Snow Labs datasets is free for personal and research purposes. For commercial use please subscribe to the [Data Library](https://www.johnsnowlabs.com/marketplace/) on John Snow Labs website. The subscription will allow you to use all John Snow Labs datasets and data packages for commercial purposes. **Included Datasets** - [Canadian Population and Dwelling by FSA 2011](https://www.johnsnowlabs.com/marketplace/canadian-population-and-dwelling-by-fsa-2011) - This Canadian Census dataset covers data on population, total private dwellings and private dwellings occupied by usual residents by forward sortation area (FSA). It is enriched with the percentage of the population or dwellings versus the total amount as well as the geographical area, province, and latitude and longitude. The whole Canada's population is marked as 100, referring to 100% for the percentages. - [Detailed Canadian Population Statistics by CMAs and CAs 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-cmas-and-cas-2011) - This dataset covers the population statistics of Canada by Census Metropolitan Areas (CMAs) and Census Agglomerations (CAs). It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. - [Detailed Canadian Population Statistics by FED 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-fed-2011) - This dataset covers the population statistics of Canada from 2011 by Federal Electoral District of 2013 Representation Order. It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. - [Detailed Canadian Population Statistics by Health Region 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-health-region-2011) - This dataset covers the population statistics of Canada by health region. It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. - [Detailed Canadian Population Statistics by Province 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-province-2011) - This dataset covers the population statistics of Canada by provinces and territories. It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. **Data Engineering Overview** **We deliver high-quality data** - Each dataset goes through 3 levels of quality review - 2 Manual reviews are done by domain experts - Then, an automated set of 60+ validations enforces every datum matches metadata & defined constraints - Data is normalized into one unified type system - All dates, unites, codes, currencies look the same - All null values are normalized to the same value - All dataset and field names are SQL and Hive compliant - Data and Metadata - Data is available in both CSV and Apache Parquet format, optimized for high read performance on distributed Hadoop, Spark & MPP clusters - Metadata is provided in the open Frictionless Data standard, and its every field is normalized & validated - Data Updates - Data updates support replace-on-update: outdated foreign keys are deprecated, not deleted **Our data is curated and enriched by domain experts** Each dataset is manually curated by our team of doctors, pharmacists, public health & medical billing experts: - Field names, descriptions, and normalized values are chosen by people who actually understand their meaning - Healthcare & life science experts add categories, search keywords, descriptions and more to each dataset - Both manual and automated data enrichment supported for clinical codes, providers, drugs, and geo-locations - The data is always kept up to date – even when the source requires manual effort to get updates - Support for data subscribers is provided directly by the domain experts who curated the data sets - Every data source’s license is manually verified to allow for royalty-free commercial use and redistribution. **Need Help?** If you have questions about our products, contact us at [info@johnsnowlabs.com](mailto:info@johnsnowlabs.com).
Databricks 收录
UCM-Captions, Sydney-Captions, RSICD, RSITMD, NWPU-Captions, RS5M, SkyScript
UCM-Captions: 包含613张图像，分辨率为256×256。Sydney-Captions: 包含2,100张图像，分辨率为500×500。RSICD: 包含10,921张图像，分辨率为224×224。RSITMD: 包含4,743张图像，分辨率为256×256。NWPU-Captions: 包含31,500张图像，分辨率为256×256。RS5M: 包含超过500万张图像，分辨率为所有可能的分辨率。SkyScript: 包含520万张图像，分辨率为所有可能的分辨率。
github 收录
QM9
QM9数据集包含134k个有机小分子化合物的量子化学计算结果，涵盖了12个量子化学性质，如分子能量、电离能、电子亲和能等。
quantum-machine.org 收录