A biodiversity dataset graph: DataONE

A biodiversity dataset graph: DataONE The intended use of this archive is to facilitate (meta-)analysis of the Data Observation Network for Earth (DataONE). DataONE is a distributed infrastructure that provides information about earth observation data. This dataset provides versioned snapshots of the DataONE network as tracked by Preston [2] between 2018-11-06 and 2020-05-07 using "preston update -u https://dataone.org". The archive consists of 256 individual parts (e.g., preston-00.tar.gz, preston-01.tar.gz, ...) to allow for parallel file downloads. The archive contains three types of files: index files, provenance logs and data files. In addition, index files have been individually included in this dataset publication to facilitate remote access. Index files provide a way to links provenance files in time to establish a versioning mechanism. Provenance files describe how, when, what and where the DataONE content was retrieved. For more information, please visit https://preston.guoda.bio or https://doi.org/10.5281/zenodo.1410543 . To retrieve and verify the downloaded DataONE biodiversity dataset graph, first concatenate all the downloaded preston-*.tar.gz files (e.g., cat preston-*.tar.gz > preston.tar.gz). Then, extract the archives into a "data" folder. Alternatively, you can use the preston[2] command-line tool to "clone" this dataset using: $ java -jar preston.jar clone --remote https://zenodo.org/record/3849494/files After that, verify the index of the archive by reproducing the following provenance log history: $ java -jar preston.jar history <0659a54f-b713-4f86-a917-5be166a14110> . . . . . . . . . . . . . . . . . . . . To check the integrity of the extracted archive, confirm that each line produce by the command "preston verify" produces lines as shown below, with each line including "CONTENT_PRESENT_VALID_HASH". Depending on hardware capacity, this may take a while. $ java -jar preston.jar verify hash://sha256/e55c1034d985740926564e94decd6dc7a70f779a33e7deb931553739cda16945 file:/home/preston/preston-dataone/data/e5/5c/e55c1034d985740926564e94decd6dc7a70f779a33e7deb931553739cda16945 OK CONTENT_PRESENT_VALID_HASH 21580 hash://sha256/e55c1034d985740926564e94decd6dc7a70f779a33e7deb931553739cda16945 hash://sha256/d0ddcc2111b6134a570bcc7d89375920ef4d754130cecc0727c79d2b05a9f81f file:/home/preston/preston-dataone/data/d0/dd/d0ddcc2111b6134a570bcc7d89375920ef4d754130cecc0727c79d2b05a9f81f OK CONTENT_PRESENT_VALID_HASH 2035 hash://sha256/d0ddcc2111b6134a570bcc7d89375920ef4d754130cecc0727c79d2b05a9f81f hash://sha256/472de9d1c9fd7e044aac409abfbfff9f12c6b69359df995d431009580ffb0f53 file:/home/preston/preston-dataone/data/47/2d/472de9d1c9fd7e044aac409abfbfff9f12c6b69359df995d431009580ffb0f53 OK CONTENT_PRESENT_VALID_HASH 1935 hash://sha256/472de9d1c9fd7e044aac409abfbfff9f12c6b69359df995d431009580ffb0f53 hash://sha256/b29879462cd43862129c5cf9b149c41ecd33ffef284a4dbea4ac1c0f90108687 file:/home/preston/preston-dataone/data/b2/98/b29879462cd43862129c5cf9b149c41ecd33ffef284a4dbea4ac1c0f90108687 OK CONTENT_PRESENT_VALID_HASH 1553 hash://sha256/b29879462cd43862129c5cf9b149c41ecd33ffef284a4dbea4ac1c0f90108687 Note that a copy of the java program "preston", preston.jar, is included in this publication. The program runs on java 8+ virtual machine using "java -jar preston.jar", or in short "preston". Files in this data publication: --- start of file descriptions --- -- description of archive and its contents (this file) -- README -- executable java jar containing preston[2] v0.1.15. -- preston.jar -- preston archives containing DataONE data files, associated provenance logs and a provenance index -- preston-[00-ff].tar.gz -- individual provenance index files -- 2a5de79372318317a382ea9a2cef069780b852b01210ef59e06b640a3539cb5a 2aecaf289def0e23a27058bf7715f226ef9189905f0be13228174825633125cf 2f65ae542401d4c2daf1bca70de640211da6749188f67d28ea71acd7d8ba070b 35eb1e17e2bf3e71212cde35bdb03e8a6545a57483ea3c1633929257b70cf637 3d38b70198e448674be6a63d14b9817f3a956f48bba7418fa7baa086a56c05b7 66ad3e5e904740f1e835ac6718dda4279e0c24b204ea0d1113cda1352a5072ba 7466a35e42dea7e2be068060ec0c926f9a8686388ed504ef5c6c990c1ba4e8d0 81161d9746c2a5823641c436e773fb4508516b055da85f4494b38c545349da39 8bf062872ce958545d361e9d53a552ffb025ac29ab875caad1157c0995d34f66 a90eed8d70c54c8e554f2dfde4fceb434eda162d9615d62de96ded2344f88a78 c33ef5e29100b323412f1f3bc66908c8e01e4f0d1db4ea3685d2fffc47981dd6 c84dffef20fec958255e759db6445fc469d73695674a33ae6f7e567a088c9fe0 d362d599d72000c4feb464db5a669b12e15fc3ca1a49b1e7d4d6f7d6d5d15411 d9378616636be3686bbabd5bf29d50f0ef0e5ceb5ddd7dfce47f7e755b596b7d da26fa6e7371385ed3f61af9a766221c833060d59dfd4869bbd7110f95f288db e4103a75627857de3ee2e317429108611c244fc448c01d1d7bf652115c3b8a55 eb368fedb8f100210dd968edcf80f4d13cab3dd64135a6ab744102cf15e68c94 f13ab4bca04f894ae8eabb51fa01b4dfbc69f717eabc9896c728e2ba39c4db27 f493baf276892a199a0b0d078359f64a38fe8ad3f807921f8d41ef73f7343b1f ff92b6c06ae5286bd2f1db679e0fcc4da294acb9bc01b2e9522378d99218c2e3 --- end of file descriptions --- References [1] Data Observation Network for Earth (DataONE, https://dataone.org) accessed from 2018-11-06 to 2020-05-07 with provenance hash://sha256/2b5c445f0b7b918c14a50de36e29a32854ed55f00d8639e09f58f049b85e50e3. [2] https://preston.guoda.bio, https://doi.org/10.5281/zenodo.1410543 . This work is funded in part by grant NSF OAC 1839201 from the National Science Foundation.