Meta-data for data.gov.uk datasets

A dataset of all the meta-data for all of the datasets available through the data.gov.uk service. This is provided as a zipped CSV or JSON file. It is published nightly.

Updates:
27 Sep 2017: we've moved all the previous dumps to an S3 bucket at https://dgu-ckan-metadata-dumps.s3-eu-west-1.amazonaws.com/ - This link is now listed here as a data file.

From 13/10/16 we added .v2.jsonl dump, which is set to replace the .json dump (which will be discontinued after a 3 month transition). This is produced using 'ckanapi dump'. It provides an enhanced version of each dataset ('validated', or what you get from package_show in CKAN API v3 - the old json was the unvalidated version). This now includes full details of the organization the dataset is in, rather than just the owner_id. Plus it includes the results of the archival & qa for each dataset and resource, showing whether the link is broken, detected format and stars of openness. It also benefits from being json lines http://jsonlines.org/ format, so you don't need to load the whole thing into memory to parse the json - just a line at a time.

On 12/1/2015 the organizations of the CSV was changed:

* Before this date, each dataset was one line, and resources added as numbered columns. Since a dataset may have up to 300 resources, it ends up with 1025 columns, which is wider than many versions of Excel and Libreoffice will open. And the uncompressed size of 170Mb is more than most will deal with too. It is suggested you load it into a database, ahandle it with a python or ruby script, or use tools such as Refine or Google Fusion Tables.

* After this date, the datasets are provided in one CSV and resources in another. On occasions that you want to join them, you can join them using the (dataset) "Name" column. These are now manageable in spreadsheet software.

You can also use the standard CKAN API if you want to search or get a small section of the data. Please respect the traffic limits in the API: http://data.gov.uk/terms-and-conditions

本数据集汇集了通过data.gov.uk服务提供的所有数据集的元数据。该数据集以压缩的CSV或JSON格式提供，并每晚进行更新。

更新记录：
27 Sep 2017：我们将所有之前的存档移动至S3存储桶，地址为https://dgu-ckan-metadata-dumps.s3-eu-west-1.amazonaws.com/。此链接现已列在此处作为数据文件。
从2016年10月13日起，我们增加了.v2.l存档，该存档将取代.存档（在三个月过渡期后将被停止使用）。该存档使用'ckanapi dump'生成，提供了每个数据集的增强版本（经过验证的，或您从CKAN API v3的package_show中获取的内容——旧的为未验证版本）。现在，它不仅包括数据集所属组织的详细信息，而非仅仅是所有者ID，还包括每个数据集和资源的存档与质量保证结果，显示链接是否损坏、检测到的格式以及开放度星级。此外，它还受益于 lines（http://lines.org/）格式，因此无需将整个内容加载到内存中即可解析，只需逐行读取即可。

2015年1月12日之前，CSV文件中的组织结构发生了变化：
* 在此日期之前，每个数据集占一行，资源作为编号列添加。由于一个数据集可能有高达300个资源，最终导致有1025列，这超出了许多Excel和Libreoffice版本的可打开范围。未压缩的大小为170Mb，也超出了大多数人的处理能力。建议您将其加载到数据库中，或使用Python或Ruby脚本进行处理，或使用Refine或Google Fusion Tables等工具。
* 此日期之后，数据集提供在单个CSV文件中，而资源提供在另一个CSV文件中。在需要将它们合并的情况下，您可以使用（数据集）“名称”列进行合并。现在这些数据集在电子表格软件中是可管理的。

如果您想要搜索或获取数据的一小部分，也可以使用标准的CKAN API。请尊重API的流量限制：http://data.gov.uk/terms-and-conditions