Awesome Public Datasets

这是一个收集和整理自互联网的公共数据源列表，涵盖了气候、经济、能源、金融、生物、物理、医疗健康、地理空间/GIS等多个领域的数据集。

This is a curated list of public data sources collected from the internet, encompassing datasets across various fields such as climate, economics, energy, finance, biology, physics, healthcare, and geospatial/GIS.

原始信息汇总

数据集概述

气候数据集

• Australian Weather: http://www.bom.gov.au/climate/dwo/
• Climate data: http://www.cru.uea.ac.uk/cru/data/temperature/ and ftp://ftp.cmdl.noaa.gov/
• Global climate data since 1929: http://www.tutiempo.net/en/Climate
• NOAA Bering Sea Climate: http://www.beringclimate.noaa.gov/
• NOAA climate datasets: http://ncdc.noaa.gov/data-access/quick-links
• WU Historical Weather Worldwide: http://www.wunderground.com/history/index.html

经济学数据集

• American Economic Ass. (AEA): http://www.aeaweb.org/RFE/toc.php?show=complete
• EconData (UMD): http://inforumweb.umd.edu/econdata/econdata.html
• Internet Product Code Database: http://www.upcdatabase.com/
• World bank: http://data.worldbank.org/indicator

能源数据集

• AMPds: http://ampds.org/
• BLUEd: http://nilm.cmubi.org/
• COMBED: http://combed.github.io/
• Dataport: https://dataport.pecanstreet.org/
• ECO: http://www.vs.inf.ethz.ch/res/show.html?what=eco-data
• iAWE: http://iawe.github.io/
• HFED: http://hfed.github.io/
• Plaid: http://plaidplug.com/
• REDD: http://redd.csail.mit.edu/
• UK-Dale: http://www.doc.ic.ac.uk/~dk3810/data/

金融数据集

• CBOE Futures Exchange: http://cfe.cboe.com/Data/
• Google Finance: https://www.google.com/finance
• Google Trends: http://www.google.com/trends?q=google&ctab=0&geo=all&date=all&sort=0
• NASDAQ: https://data.nasdaq.com/
• OANDA: http://www.oanda.com/
• OSU Financial data: http://fisher.osu.edu/fin/osudata.htm or http://fisher.osu.edu/fin/fdf/osudata.htm
• Quandl: http://www.quandl.com/
• St Louis Federal: http://research.stlouisfed.org/fred2/
• Yahoo Finance: http://finance.yahoo.com/

生物学数据集

• CRCNS: http://crcns.org/data-sets
• Gene Expression Omnibus: http://www.ncbi.nlm.nih.gov/geo/
• Human Microbiome Project: http://www.hmpdacc.org/reference_genomes/reference_genomes.php
• MIT Cancer Genomics Data: http://www.broadinstitute.org/cgi-bin/cancer/datasets.cgi
• NIH Microarray data: ftp://ftp.ncbi.nih.gov/pub/geo/DATA/supplementary/series/GSE6532/
• Protein structure: http://www.infobiotic.net/PSPbenchmarks/
• Protein Data Bank: http://pdb.org/
• Public Gene Data: http://www.pubgene.org/
• Stanford Microarray Data: http://smd.stanford.edu/
• UniGene: http://www.ncbi.nlm.nih.gov/unigene
• The Personal Genome Project: http://www.personalgenomes.org/ or https://my.pgp-hms.org/public_genetic_data
• 1000 Genomes: http://www.1000genomes.org/data
• UCSC Public Data: http://hgdownload.soe.ucsc.edu/downloads.html

物理学数据集

• NASA: http://nssdc.gsfc.nasa.gov/nssdc/obtaining_data.html

医疗保健数据集

• EHDP Large Health Data Sets: http://www.ehdp.com/vitalnet/datasets.htm
• Gapminder: http://www.gapminder.org/data/
• Medicare Data File: http://go.cms.gov/19xxPN4

地理空间/GIS数据集

• EOSDIS: http://sedac.ciesin.columbia.edu/data/sets/browse
• Factual Global Location Data: http://www.factual.com/
• Geo Spatial Data: http://geodacenter.asu.edu/datalist/
• OpenStreetMap: http://wiki.openstreetmap.org/wiki/Downloading_data
• GeoNames: http://www.geonames.org/
• BODC: http://www.bodc.ac.uk/data/where_to_find_data/
• GADM: http://www.gadm.org/
• twofishes: https://github.com/foursquare/twofishes
• Natural Earth: http://www.naturalearthdata.com/
• tz_world: http://efele.net/maps/tz/world/
• TIGER/Line: http://www.census.gov/geo/maps-data/data/tiger-line.html

交通运输数据集

• Airlines Data (2009 ASA Challenge): http://stat-computing.org/dataexpo/2009/the-data.html
• Bike Share Data Systems: https://github.com/BetaNYC/Bike-Share-Data-Best-Practices/wiki/Bike-Share-Data-Systems
• Edge data for US domestic flights 1990 to 2009: http://data.memect.com/?p=229
• Half a million Hubway rides: http://hubwaydatachallenge.org/trip-history-data/
• NYC Taxi Trip Data 2013 (FOIA/FOIL): https://archive.org/details/nycTaxiTripData2013
• OpenFlights: http://openflights.org/data.html
• RITA Airline On-Time Performance Data: http://www.transtats.bts.gov/Tables.asp?DB_ID=120
• RITA transport data collection: http://www.transtats.bts.gov/DataIndex.asp
• Transport for London: http://www.tfl.gov.uk/info-for/open-data-users/our-feeds
• U.S. Freight Analysis Framework: http://ops.fhwa.dot.gov/freight/freight_analysis/faf/index.htm
• Marine Traffic: https://www.marinetraffic.com/de/p/api-services

政府数据集

• Archive-it: https://www.archive-it.org/explore?show=Collections
• Australia: https://data.gov.au/
• Australia: http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/3301.02009?OpenDocument
• Canada: http://www.data.gc.ca/default.asp?lang=En&n=5BCD274E-1
• Chicago: https://data.cityofchicago.org/
• FDA: https://open.fda.gov/index.html
• Fed Stats: http://www.fedstats.gov/cgi-bin/A2Z.cgi
• Guardian world governments: http://www.guardian.co.uk/world-government-data
• HUD: http://www.huduser.org/portal/datasets/pdrdatas.html
• London Datastore, U.K: http://data.london.gov.uk/dataset
• Glasgow, Scotland, UK: http://data.glasgow.gov.uk/
• Netherlands: https://data.overheid.nl/
• New Zealand: http://www.stats.govt.nz/browse_for_stats.aspx
• NYC betanyc: http://betanyc.us/
• NYC Open Data: http://nycplatform.socrata.com/
• OECD: http://www.oecd.org/document/0,3746,en_2649_201185_46462759_1_1_1_1,00.html
• RITA: http://www.transtats.bts.gov/OT_Delay/OT_DelayCause1.asp
• San Francisco Data sets: http://datasf.org/
• The World Bank: http://wdronline.worldbank.org/
• U.K. Government Data: http://data.gov.uk/data
• U.S. Census Bureau: http://www.census.gov/data.html
• U.S. American Community Survey: http://www.census.gov/acs/www/data_documentation/data_release_info/
• U.S. Federal Government Agencies: http://www.data.gov/metric
• U.S. Federal Government Data Catalog: http://catalog.data.gov/dataset
• U.S. Open Government: http://www.data.gov/open-gov/
• UK 2011 Census Open Atlas Project: http://www.alex-singleton.com/2011-census-open-atlas-project/
• United Nations: http://data.un.org/
• US CDC Public Health datasets: http://www.cdc.gov/nchs/data_access/ftp_data.htm
• Open Government Data (OGD) Platform India: http://www.data.gov.in/

数据挑战

• Challenges in Machine Learning: http://www.chalearn.org/
• ICWSM Data Challenge (since 2009): http://icwsm.cs.umbc.edu/
• Kaggle Competition Data: http://www.kaggle.com/
• KDD Cup by Tencent 2012: https://www.kddcup2012.org/
• Netflix Prize: http://www.netflixprize.com/leaderboard
• Yelp Dataset Challenge: http://www.yelp.com/dataset_challenge

机器学习数据集

• eBay Online Auctions: http://www.modelingonlineauctions.com/datasets
• IMDb database: http://www.imdb.com/interfaces
• Keel Repository: http://sci2s.ugr.es/keel/datasets.php
• Lending Club Loan Data: https://www.lendingclub.com/info/download-data.action
• Machine Learning Data Set Repository: http://mldata.org/
• Million Song Dataset: http://blog.echonest.com/post/3639160982/million-song-dataset
• More Song Datasets: http://labrosa.ee.columbia.edu/millionsong/pages/additional-datasets
• MovieLens Data Sets: http://datahub.io/dataset/movielens
• RDataMining R and Data Mining ebook data: http://www.rdatamining.com/data
• Registered meteorites on Earth: http://www.analyticbridge.com/profiles/blogs/registered-meteorites-that-has-impacted-on-earth-visualized
• SF restaurants dataset: http://missionlocal.org/san-francisco-restaurant-health-inspections/
• UCI Machine Learning Repository: http://archive.ics.uci.edu/ml/
• University of Toronto Delve Datasets: http://www.cs.toronto.edu/~delve/data/datasets.html
• Yahoo Ratings and Classification Data: http://webscope.sandbox.yahoo.com/catalog.php?datatype=r

自然语言数据集

• 40 Million Entities in Context: https://code.google.com/p/wiki-links/downloads/list
• ClueWeb09 FACC: http://lemurproject.org/clueweb09/FACC1/
• ClueWeb12 FACC: http://lemurproject.org/clueweb12/FACC1/
• Flickr personal taxonomies: http://www.isi.edu/~lerman/downloads/flickr/flickr_taxonomies.html
• Google Books Ngrams: http://aws.amazon.com/datasets/8172056142375670
• Google Web 5gram, 2006 (1T): https://catalog.ldc.upenn.edu/LDC2006T13
• Gutenberg eBooks List: http://www.gutenberg.org/wiki/Gutenberg:Offline_Catalogs
• Hansards: http://www.isi.edu/natural-language/download/hansard/
• Machine Translation: http://statmt.org/wmt11/translation-task.html#download
• SMS Spam Collection: http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/
• USENET corpus: http://www.psych.ualberta.ca/~westburylab/downloads/usenetcorpus.download.html
• WordNet: http://wordnet.princeton.edu/wordnet/download/

图像处理数据集

• 2GB of photos of cats: http://137.189.35.203/WebUI/CatDatabase/catData.html
• Face Recognition Benchmark: http://www.face-rec.org/databases/
• ImageNet: http://www.image-net.org/

时间序列数据集

• Time Series data Library: https://datamarket.com/data/list/?q=provider:tsdl
• UC Riverside Time Series: http://www.cs.ucr.edu/~eamonn/time_series_data/

社会科学数据集

• China Hotel Checkin/out data: http://www.360doc.com/content/13/1105/13/7863900_326788919.shtml
• CMU Enron Email: http://www.cs.cmu.edu/~enron/
• Facebook Social Networks (since 2007): http://law.di.unimi.it/datasets.php
• Facebook100 (2005): https://archive.org/details/oxford-2005-facebook-matrix
• Foursquare (2010,2011): http://www.public.asu.edu/~hgao16/dataset.html
• Foursquare (UMN/Sarwat, 2013): https://archive.org/details/201309_foursquare_dataset_umn
• General Social Survey (GSS): http://www3.norc.org/GSS+Website/
• GetGlue (users rating TV shows): http://bit.ly/1aL8XS0
• GitHub Archive: http://www.githubarchive.org/
• ICPSR: http://www.icpsr.umich.edu/icpsrweb/ICPSR/index.jsp
• Mobile Social Networks (UMASS): https://kdl.cs.umass.edu/display/public/Mobile+Social+Networks
• PewResearch Internet Project: http://www.pewinternet.org/datasets/pages/2/
• Social Networking: http://www.cs.cmu.edu/~jelsas/data/ancestry.com/
• SourceForge Graph: http://www.nd.edu/~oss/Data/data.html
• Titanic Survival Data Set: https://github.com/caesar0301/awesome-public-datasets/blob/master/Datasets/titanic.csv.zip
• Twitter Graph: http://an.kaist.ac.kr/traces/WWW2010.html
• UC Berkeleys D-Lab Achive: http://ucdata.berkeley.edu/
• UCLA Social Sciences Data Archive: http://dataarchives.ss.ucla.edu/Home.DataPortals.htm
• UNIMI Social Network Datasets: http://law.di.unimi.it/datasets.php
• Universities Worldwide: http://univ.cc/
• UPJOHN for Employment Research: http://www.upjohn.org/erdc/erdc.html
• Yahoo Graph and Social Data: http://webscope.sandbox.yahoo.com/catalog.php?datatype=g
• **Youtube Graph (2

搜集汇总

数据集介绍

构建方式

Awesome Public Datasets 数据集通过广泛收集和整理来自博客、问答平台及用户反馈的公开数据源构建而成。其涵盖了多个领域，包括气候、经济、能源、金融、生物学、物理学、医疗保健、地理空间、交通运输、政府数据、机器学习、自然语言处理、图像处理、时间序列、社会科学、复杂网络、计算机网络、博物馆数据等。数据集的构建过程注重数据的多样性和广泛性，确保每个领域的数据源都具有代表性。

特点

该数据集的特点在于其广泛的覆盖范围和多样化的数据来源。它不仅包含了来自政府机构、科研机构和企业的数据，还涵盖了多个学科领域，能够满足不同研究需求。数据集中的大部分数据是免费的，但也有部分数据需要付费获取。此外，数据集中的数据源经过筛选和整理，确保了数据的可靠性和可用性。

使用方法

使用 Awesome Public Datasets 数据集时，用户可以根据研究需求选择相应的数据源。数据集提供了每个数据源的链接和简要描述，用户可以直接访问这些链接获取数据。对于需要进一步处理的数据，用户可以根据数据格式和内容进行清洗、分析和建模。该数据集特别适合用于跨学科研究、数据科学项目以及机器学习模型的训练和验证。用户还可以通过数据集提供的链接探索更多相关资源，以扩展研究范围。

背景与挑战

背景概述

Awesome Public Datasets 是一个广泛收集和整理公共数据源的资源库，涵盖了从气候、经济、能源到生物学、物理学、医疗保健等多个领域的数据集。该数据集由GitHub用户caesar0301于2014年创建，旨在为研究人员、数据科学家和开发者提供一个便捷的平台，以获取高质量、多样化的公共数据。其核心研究问题在于如何有效地整合和分类来自不同领域的数据，以便于用户快速找到所需资源。该数据集的影响力不仅体现在其广泛的数据覆盖范围，还在于其推动了数据共享和开放科学的发展，为跨学科研究提供了重要支持。

当前挑战

Awesome Public Datasets 面临的主要挑战包括数据集的多样性和复杂性。首先，由于数据集涵盖的领域广泛，如何确保数据的准确性和一致性成为一个关键问题。其次，数据集的更新和维护需要持续投入，以确保数据的时效性和可用性。此外，数据集的构建过程中，如何有效地整合来自不同来源的数据，并解决数据格式、标准和元数据的差异，也是一个技术难题。最后，随着数据量的增加，如何优化数据检索和访问效率，以满足用户需求，也是该数据集需要持续改进的方向。

常用场景

经典使用场景

Awesome Public Datasets 数据集广泛应用于学术研究和工业界的数据分析项目中。该数据集涵盖了气候、经济、能源、金融、生物学、物理学、医疗保健、地理空间、交通运输、政府、数据挑战、机器学习、自然语言处理、图像处理、时间序列、社会科学、复杂网络、计算机网络、博物馆等多个领域。研究人员和开发者可以通过该数据集获取高质量、多样化的公开数据，用于模型训练、算法验证和数据分析。

解决学术问题

Awesome Public Datasets 解决了学术研究中数据获取难、数据质量参差不齐的问题。通过整合全球范围内的公开数据集，研究人员可以轻松访问到经过整理和验证的数据，从而专注于研究问题的核心。例如，在气候研究中，该数据集提供了全球气候数据，帮助科学家分析气候变化趋势；在生物学领域，基因表达和蛋白质结构数据为生物信息学研究提供了重要支持。

衍生相关工作

Awesome Public Datasets 衍生了许多经典的研究工作。例如，基于该数据集中的气候数据，研究人员发表了多篇关于全球变暖的论文；在机器学习领域，UCI Machine Learning Repository 中的数据被广泛用于算法开发和模型评估；在自然语言处理领域，Google Books Ngrams 数据为语言模型的研究提供了重要支持。此外，该数据集还催生了许多开源工具和平台，进一步推动了数据科学的发展。