awesome-public-datasets

这是一个包含高质量公开数据集的列表，涵盖了农业、生物学等多个领域的数据集，这些数据集主要来源于公共领域，且大多数是免费的。

This is a list of high-quality public datasets covering various fields such as agriculture and biology. These datasets are primarily sourced from the public domain and are mostly free of charge.

原始信息汇总

数据集概述

本数据集名为“Awesome Public Datasets”，收集并整理了来自博客、问答和用户响应的公共数据源。大部分数据集是免费的，但也有一些并非免费。数据集涵盖了多个领域，包括农业、生物学、气候/天气、复杂网络、计算机网络、上下文数据、数据挑战、经济学、教育、能源、金融、地质、GIS/环境、政府、医疗保健、图像处理、机器学习、博物馆、自然语言等。

数据集分类及来源

农业

• U.S. Department of Agricultures PLANTS Database

生物学

• 1000 Genomes
• American Gut (Microbiome Project)
• Broad Cancer Cell Line Encyclopedia (CCLE)
• Broad Bioimage Benchmark Collection (BBBC)
• Cell Image Library
• Collaborative Research in Computational Neuroscience (CRCNS)
• Complete Genomics Public Data
• EBI ArrayExpress
• EBI Protein Data Bank in Europe
• Electron Microscopy Pilot Image Archive (EMPIAR)
• ENCODE project
• Ensembl Genomes
• Gene Expression Omnibus (GEO)
• Gene Ontology (GO)
• Global Biotic Interactions (GloBI)
• Harvard Medical School (HMS) LINCS Project
• Human Genome Diversity Project
• Human Microbiome Project (HMP)
• ICOS PSP Benchmark
• International HapMap Project
• Journal of Cell Biology DataViewer
• MIT Cancer Genomics Data
• NCBI Proteins
• NCBI Taxonomy
• NeuroData
• NIH Microarray data
• OpenSNP genotypes data
• Pathguid - Protein-Protein Interactions Catalog
• Protein Data Bank
• Psychiatric Genomics Consortium
• PubChem Project
• PubGene (now Coremine Medical)
• Sanger Catalogue of Somatic Mutations in Cancer (COSMIC)
• Sanger Genomics of Drug Sensitivity in Cancer Project (GDSC)
• Sequence Read Archive(SRA)
• Stanford Microarray Data
• Stowers Institute Original Data Repository
• Systems Science of Biological Dynamics (SSBD) Database
• Temple University Hospital EEG Database
• The Cancer Genome Atlas (TCGA), available via Broad GDAC
• The Catalogue of Life
• The Personal Genome Project
• UCSC Public Data
• Universal Protein Resource (UnitProt)
• UniGene

气候/天气

• Australian Weather
• Brazilian Weather - Historical data (In Portuguese)
• Canadian Meteorological Centre
• Climate Data from UEA (updated monthly)
• European Climate Assessment & Dataset
• Global Climate Data Since 1929
• NASA Global Imagery Browse Services
• NOAA Bering Sea Climate
• NOAA Climate Datasets
• NOAA Realtime Weather Models
• The World Bank Open Data Resources for Climate Change
• UEA Climatic Research Unit
• WorldClim - Global Climate Data
• WU Historical Weather Worldwide

复杂网络

• AMiner Citation Network Dataset
• CrossRef DOI URLs
• DBLP Citation dataset
• NBER Patent Citations
• Network Repository with Interactive Exploratory Analysis Tools
• NIST complex networks data collection
• Protein-protein interaction network
• PyPI and Maven Dependency Network
• Scopus Citation Database
• Small Network Data
• Stanford GraphBase (Steven Skiena)
• Stanford Large Network Dataset Collection
• Stanford Longitudinal Network Data Sources
• The Koblenz Network Collection
• The Laboratory for Web Algorithmics (UNIMI)
• The Nexus Network Repository
• UCI Network Data Repository
• UFL sparse matrix collection
• WSU Graph Database
• DIMACS Road Networks Collection

计算机网络

• 3.5B Web Pages from CommonCraw 2012
• 53.5B Web clicks of 100K users in Indiana Univ.
• CAIDA Internet Datasets
• ClueWeb09 - 1B web pages
• ClueWeb12 - 733M web pages
• CommonCrawl Web Data over 7 years
• CRAWDAD Wireless datasets from Dartmouth Univ.
• Criteo click-through data
• Open Mobile Data by MobiPerf
• Rapid7 Sonar Internet Scans
• UCSD Network Telescope, IPv4 /8 net

上下文数据

• Context-aware data sets from five domains

数据挑战

• Challenges in Machine Learning
• CrowdANALYTIX dataX
• D4D Challenge of Orange
• DrivenData Competitions for Social Good
• ICWSM Data Challenge (since 2009)
• Kaggle Competition Data
• KDD Cup by Tencent 2012
• Localytics Data Visualization Challenge
• Netflix Prize
• Space Apps Challenge
• Telecom Italia Big Data Challenge
• Yelp Dataset Challenge
• Bruteforce Database

经济学

• American Economic Ass (AEA)
• EconData from UMD
• Economic Freedom of the World Data
• Historical MacroEconomic Statistics
• International Trade Statistics
• Internet Product Code Database
• Joint External Debt Data Hub
• Jon Haveman International Trade Data Links
• OpenCorporates Database of Companies in the World
• Our World in Data
• SciencesPo World Trade Gravity Datasets
• The Atlas of Economic Complexity
• The Center for International Data
• The Observatory of Economic Complexity
• UN Commodity Trade Statistics
• UN Human Development Reports

教育

• Student Data from Free Code Camp

能源

• AMPds
• BLUEd
• COMBED
• Dataport
• ECO
• EIA
• HFED
• iAWE
• Plaid
• REDD
• UK-Dale

金融

• CBOE Futures Exchange
• Google Finance
• Google Trends
• NASDAQ
• OANDA
• OSU Financial data
• Quandl
• St Louis Federal
• Yahoo Finance
• NYSE Market Data

地质

• Earth Models
• Smithsonian Institution Global Volcano and Eruption Database
• USGS Earthquake Archives

GIS/环境

• BODC - marine data of ~22K vars
• Cambridge, MA, US, GIS data on GitHub
• EOSDIS - NASAs earth observing system data
• Factual Global Location Data
• Geo Spatial Data from ASU
• Geo Wiki Project - Citizen-driven Environmental Monitoring
• GeoFabrik - OSM data extracted to a variety of formats and areas
• GeoNames Worldwide
• Global Administrative Areas Database (GADM)
• Homeland Infrastructure Foundation-Level Data
• Integrated Marine Observing System (IMOS) - roughly 30TB of ocean measurements
• International Institute for Systems Analysis - GIS Datasets
• Landsat 8 on AWS
• List of all countries in all languages
• Marinexplore - Open Oceanographic Data
• National Weather Service GIS Data Portal
• Natural Earth - vectors and rasters of the world
• OpenAddresses
• OpenStreetMap (OSM)
• Pleiades - Gazetteer and graph of ancient places
• Reverse Geocoder using OSM data
• TIGER/Line - U.S. boundaries and roads
• TwoFishes - Foursquares coarse geocoder
• TZ Timezones shapfiles
• UN Environmental Data
• World boundaries from the U.S. Department of State
• World countries in multiple formats

政府

• OpenDataSofts list of 1,600 open data portals
• A list of cities and countries contributed by community

医疗保健

• EHDP Large Health Data Sets
• Gapminder World demographic databases
• Medicare Coverage Database (MCD), U.S.
• Medicare Data Engine of medicare.gov Data
• Medicare Data File
• MeSH, the vocabulary thesaurus used for indexing articles for PubMed
• Number of Ebola Cases and Deaths in Affected Countries (2014)
• Open-ODS (structure of the UK NHS)
• OpenPaymentsData, Healthcare financial relationship data
• The Cancer Genome Atlas project (TCGA)
• World Health Organization Global Health Observatory

图像处理

• 10k US Adult Faces Database
• 2GB of Photos of Cats
• Affective Image Classification
• Animals with attributes
• Face Recognition Benchmark
• ImageNet (in WordNet hierarchy)
• Indoor Scene Recognition
• International Affective Picture System, UFL
• Massive Visual Memory Stimuli, MIT
• Several Shape-from-Silhouette Datasets
• Stanford Dogs Dataset
• SUN database, MIT
• The Oxford-IIIT Pet Dataset
• YouTube Faces Database
• Adience Unfiltered faces for gender and age classification
• The Action Similarity Labeling (ASLAN) Challenge
• Violent-Flows - Crowd Violence Non-violence Database and benchmark

机器学习

• Delve Datasets for classification and regression (Univ. of Toronto)
• Discogs Monthly Data
• eBay Online Auctions (2012)
• IMDb Database
• Keel Repository for classification, regression and time series
• Labeled Faces in the Wild (LFW)
• Lending Club Loan Data
• Machine Learning Data Set Repository
• Million Song Dataset
• More Song Datasets
• MovieLens Data Sets
• RDataMining - "R and Data Mining" ebook data
• Registered Meteorites on Earth
• Restaurants Health Score Data in San Francisco
• UCI Machine Learning Repository
• Yahoo! Ratings and Classification Data

博物馆

• Canada Science and Technology Museums Corporations Open Data
• Cooper-Hewitts Collection Database
• Minneapolis Institute of Arts metadata
• Natural History Museum (London) Data Portal
• Rijksmuseum Historical Art Collection
• Tate Collection metadata
• The Getty vocabularies

自然语言

• Blogger Corpus
• CLiPS Stylometry Investigation Corpus
• ClueWeb09

搜集汇总

数据集介绍

构建方式

该数据集是通过从博客、回答和用户响应中收集和整理公共数据源而构建的。数据集的构建目的是为了提供一个关于公共数据集的详尽列表，其中大部分数据集是免费的，但也有一部分数据集不是免费的。

特点

该数据集的特点在于它涵盖了多个领域，包括农业、生物学、气候/天气、复杂网络、计算机网络、上下文数据、数据挑战、经济学、教育、能源、金融、地质学、GIS/环境、政府、医疗保健、图像处理、机器学习、博物馆和自然语言处理等。此外，它还包含了各种类型的数据，如文本、图像、数值等。

使用方法

使用该数据集的方法包括：首先，访问GitHub页面上的README文件以了解数据集的背景和内容；其次，根据需要浏览和选择合适的数据集；最后，根据数据集的许可和使用条款进行相应的研究或应用。

背景与挑战

背景概述

awesome-public-datasets 数据集是一个收集自博客、回答和用户响应的公共数据源列表。该数据集的创建时间为未知，主要研究人员或机构为caesar0301。该数据集的核心研究问题是搜集和整理免费的公共数据集，并对相关领域产生了积极的影响，为研究者和开发者提供了丰富的数据资源。

当前挑战

该数据集在构建过程中所遇到的挑战主要包括：1) 数据集的收集和整理依赖于社区贡献，可能存在数据质量参差不齐的问题；2) 部分数据集可能不是免费的，这给数据集的广泛应用带来限制；3) 数据集的多样性和覆盖领域广泛，可能导致领域特定的问题和挑战；4) 缺乏统一的数据格式和标准，可能增加数据集整合和处理的难度。

常用场景

经典使用场景

awesome-public-datasets数据集广泛收集了各领域的公共数据集，其经典使用场景主要在于为研究人员提供便捷的数据集搜索和整理服务。用户可以通过该数据集快速定位到所需领域的公共数据资源，从而高效地开展数据分析和研究工作。

实际应用

在实际应用中，awesome-public-datasets数据集可用于教育、科研、商业分析等多个领域。例如，教育工作者可以利用其中的数据集进行教学演示；科研人员可以基于这些数据集开展研究；商业分析师可以运用数据集进行市场分析和预测。

衍生相关工作

基于awesome-public-datasets数据集，衍生出了一系列相关工作。包括但不限于数据集的整理、数据分析框架的构建以及基于特定数据集的学术研究等。这些工作进一步推动了相关领域的研究进展和数据应用。