Zenodo metadata JSON records as of 2019-09-16

This preliminary dataset contains the <code>application/vnd.zenodo.v1+json</code> JSON records of Zenodo deposits as retrieved on 2019-09-16. <strong>Files</strong> <code>zenodo-records-json-2019-09-16.tar.xz</code> <em>Zenodo JSON records</em><br> XZ-compressed tar archive of individual JSON records as retrieved from Zenodo. Filenames reflects record, e.g. <code>1310621.json</code> was retrieved from https://zenodo.org/api/records/1310621 using content-negotiation for <code>application/vnd.zenodo.v1+json</code> <code>zenodo-records-json-2019-09-16-filtered.jsonseq.xz</code> <em>Concatinated Zenodo JSON records</em><br> XZ-compressed RFC7464 JSON Sequence stream, readable by jq. Concatination of Zenodo JSON records. Order not significant. <code>zenodo-records.sh</code> <em>Retrieve Zenodo JSON records</em><br> A retrospectively created Bash shell script that shows the commands used to retrieve JSON files and concationate to jsonseq. <code>ro-crate-metadata.jsonld</code> <em>RO-Crate 0.2 structured metadata</em> <code>ro-crate-preview.html</code> <em>Browser rendering of RO-Crate structured metadata</em> <code>README.md</code> <em>This dataset description</em> <strong>License</strong> This dataset is provided under the license Apache License, version 2.0: Copyright 2019 The University of Manchester Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at <pre><code>http://www.apache.org/licenses/LICENSE-2.0 </code></pre> Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. <strong>CC0 for Zenodo metadata</strong> The Zenodo metadata in <code>zenodo-records-json-2019-09-16.tar.xz</code> is reused under the terms of https://creativecommons.org/publicdomain/zero/1.0/ <strong>Reproducibility</strong> To retrieve the Zenodo JSON it was deemed necessary to use the undocumented parts of Zenodo API. From the Zenodo source code it was identified that the REST template <code>https://zenodo.org/api/records/{pid_value}</code> could be used with <code>pid_value</code> as the numeric part from the OAI-PMH identifier, e.g. for <code>oai:zenodo.org:1310621</code> the Zenodo JSON can be retrieved at https://zenodo.org/api/records/1310621. The JSON API supports content negotiation, the content-types supported as of 2019-09-20 include: <code>application/vnd.zenodo.v1+json</code> giving the Zenodo record in Zenodo's internal JSON schema (v1) <code>application/ld+json</code> giving JSON-LD Linked Data using the http://schema.org/ vocabulary <code>application/x-datacite-v41+xml</code> giving DataCite v4 XML <code>application/marcxml+xml</code> giving MARC 21 XML Using these (currently) undocumented parts of the Zenodo API thus avoids the need for HTML scraping while also giving individual complete records that are suitable to redistribute as records in a filtered dataset. This preliminary exploration will be adapted into the reproducible CWL workflow, for now included as a Bash script <code>zenodo-records.sh</code> Execution time was about 3 days from a server at the University of Manchester network on a single 1 GBps network link. The script does: Retrieve each of the first 3.5 million Zenodo records<br> as Zenodo JSON by iterating over possible numeric IDs (the maximum ID <code>3450000</code> was estimated from "Recent uploads") Filter list to exclude records that are not found, moved or deleted. The presence of the key <code>conceptrecid</code> is used as marker. Use jq to ensure the JSON is on a single line Join the JSON files using the ASCII Record Separator (RS, <code>0x1e</code>) to make a <code>application/json-seq</code> JSON text sequence stream Save the JSON stream as a single compressed file using <code>xz</code>