Catalogue of Life Repackaged and Sorted hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd hash://md5/882b8744b3ebd5fae371fa659ee52a2b
收藏资源简介：
Taxonomic name alignment is a necessary, and often time consuming, task when integration biodiversity datasets for re-use. This publication aims to facilitate fast (e.g., > 1k names/s), offline-enabled, and reproducible, name alignment workflows through the Catalogue of Life by repackaging optimized versions of the wealth of names contained in the Catalogue of Life. <strong>Introduction</strong> The Catalogue of Life (Bánki 2023) is “[…] is an assembly of expert-based global species checklists with the aim to build a comprehensive catalogue of all known species of organisms on Earth. […]” This data publication contains a verifiable copy of the Catalogue of Life as well as a reverse sorted version of the NameUsage.tsv table. The aims of this publication are to: provide a signed citation (Elliott, Poelen, and Fortes 2023) for a copy of Catalogue of Life prepare Catalogue of Life to be included in the Nomer Corpus of Taxonomic Resources (J. H. (ed. ). Poelen 2023). pre-process the Catalogue of Life to facilitate optimized indexing and offline taxonomic name alignments using tools like Nomer (J. Poelen and Salim 2023). Overall, the publications aims to facilitate taxonomic name alignment using the wealth of information provided by the Catalogue of Life to help enable fast, reproduceable, offline-enabled alignment of namelists with taxonomic resources of known provenance (or origin). An example of an application facilitated by this publication is the Taxonomic Name Alignment tool as provided through https://github.com/globalbioticinteractions/name-alignment-template. This template repository implements an automated workflow using GitHub Action to align scientific names in csv/tsv files and darwin core archive with common taxonomic name lists like Catalogue of Life, NCBI Taxonomy, Integrated Taxonomic Information System (ITIS), and GBIF Backbone taxonomy. <strong>Methods</strong> To capture and process the Catalogue of Life, the following steps were taken: track and archive a copy of Catalogue of Life reverse sort NameUsage.tsv assign an alias to the processed resources Steps 1-3 are captured and documented using Preston, a biodiversity data tracker. Preston not only helps to documents the steps, but also includes the digital resources that were used and produced. Track and Archive To track and archive a copy of Catalogue of Life, the following command was issued: <pre><code>preston track https://download.catalogueoflife.org/col/latest_coldp.zip</code></pre> With this, a copy of https://download.catalogueoflife.org/col/latest_coldp.zip is downloaded and their sha256 checksum (or hash) is calculated. Also, the download process is captured machine-readable rdf/nquads statement. The content id (or sha256 hash) of the copy included in this publication can be found using: <pre><code>preston alias\ --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd\ --remote https://zenodo.org/deposit/8327611/files\ https://download.catalogueoflife.org/col/latest_coldp.zip\ head -n1</code></pre> and is <pre><code><https://download.catalogueoflife.org/col/latest_coldp.zip> <http://purl.org/pav/hasVersion> <hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1> <urn:uuid:f00d82a2-d965-4b1d-9030-0a2f8833e004> .</code></pre> meaning that <code>hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1</code> is the content id (or sha256 hash) of the content produced by <code>https://download.catalogueoflife.org/col/latest_coldp.zip</code> at the time this publication was compiled. Reverse Sort NameUsage.tsv The second step was to pre-process (or transform) part of the tracked Catalogue of Life data package and reverse sort their content. We use the following bash script to do this processing: <pre><code>(sed -u 1q; LC_ALL=C sort -r) | gzip</code></pre> The script first prints the header (i.e., <code>sed -u 1q</code>), then reverse sorts the following content (i.e., <code>LC_ALL=C sort -r</code>). Finally, the output is compressed using gzip. This script is part of this data publication, and can be retrieved via: <pre><code>preston cat\ --remote https://zenodo.org/record/8327611/files/\ hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd</code></pre> To help make it easier to reference the script, an alias was created using <pre><code>preston alias\ urn:example:reverse-sort.sh\ hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd ~~ Following the authoring of the reverse sort script, and documenting their alias (i.e. 'urn:example:reverse-sort.sh`), we applied the script to the acquired copy of the Catalogue of Life using: </code></pre> preston cat<br> –remote https://zenodo.org/record/8327611/files/<br> ‘zip:hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1!/NameUsage.tsv’<br> | preston bash<br> –anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd<br> –remote https://zenodo.org/record/8327611/files/<br> -c urn:example:reverse-sort.sh ~~~ The result of this process was the content identified by sha256 hash <pre><code>hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc</code></pre> as documented in line 22 of the associated preston processing log in <pre><code>preston cat\ --remote https://zenodo.org/record/8327611/files/\ 'line:hash://sha256/8ba35deafc847f0d5d69d357241a431b7fd9b6f2735189575b2b7168d523caa9!/L22' </code></pre> Assign An Alias to Processed Resources The alias ‘col:NameUsage.tsv.gz’ was defined to help make it easier to point to the result using: <pre><code>preston alias\ col:NameUsage.tsv.gz\ hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc</code></pre> With this, the following command was executed to list the first three lines of the produced resource: <pre><code>preston cat\ --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd\ --remote https://zenodo.org/record/8327611/files/\ col:NameUsage.tsv.gz\ | gunzip\ | head -n3</code></pre> Where <code>preston cat ...</code> prints the produced resource, <code>gunzip</code> uncompresses the result, and the first three lines are selected using <code>head -n3</code>. The result of the operation is shown below: col:ID col:alternativeID col:nameAlternativeID col:sourceID col:parentID col:basionymID col:status col:scientificName col:authorship col:rank col:notho col:uninomial col:genericName col:infragenericEpithet col:specificEpithet col:infraspecificEpithet col:cultivarEpithet col:namePhrase col:nameReferenceID col:publishedInYear col:publishedInPage col:publishedInPageLink col:code col:nameStatus col:accordingToID col:accordingToPage col:accordingToPageLink col:referenceID col:scrutinizer col:scrutinizerID col:scrutinizerDate col:extinct col:temporalRangeStart col:temporalRangeEnd col:environment col:species col:section col:subgenus col:genus col:subtribe col:tribe col:subfamily col:family col:superfamily col:suborder col:order col:subclass col:class col:subphylum col:phylum col:kingdom col:sequenceIndex col:branchLength col:link col:nameRemarks col:remarks ffc77d7d-2ede-49ff-ab12-03410a1c25db 55434 93MTR 4WGYN provisionally accepted [Semiothisa] lapidata Warren, 1906 species zoological acceptable false ff82a38f-348b-4fd7-891e-a2a68d8edce4 55434 93MTR 78YFQ provisionally accepted [Sabulodes] arnissa Druce, 1891 species zoological acceptable false <strong>Results</strong> As described in our methods, this publication derived the resource with alias col:NameUsage.tsv.gz and content id <code>hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc</code> . This resource contains a reverse-sorted copy of the NameUsage.tsv provided in the Catalogue of Life data package retrieved from https://download.catalogueoflife.org/col/latest_coldp.zip with content identifier <code>hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1</code>. The following tools were used to process the Catalogue of Life resource: Tools used in this data publication tool name preston bash gzip sed head sort <strong>Discussion</strong> This publication is intended to facilitate re-use of the Catalogue of Life data package in taxonomic name alignment workflows. While the primary goal was to generate a resource for use in Nomer v0.4.5 (J. Poelen and Salim 2023), other usage can be imagined such as: Lots of Copies Keeps Stuff Safe (LOCKSS (Maniatis et al. 2005)): keep an identical copy of Catalogue of Life data package outside of the Catalogue of Life infrastructure. demonstrating how data transformation processes can be documented using Preston making a streamable copy of a reverse-sorted copy of Catalogue of Life available via https://zenodo.org/record/8327611/files/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc for use in workflows like looking up the first record that contain <em>Enhydra lutris</em> (Sea otter): <pre><code> curl -L 'https://zenodo.org/record/8327611/files/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc'\ | gunzip\ | grep "Enhydra lutris"\ | head -n1</code></pre> <strong>Acknowledgements</strong> This work stands on the shoulders of contributors to open source software and openly accessible datasets. Thank you! <strong>References</strong> Bánki, O et al. 2023. “Catalogue of Life Checklist (Version 2023-08-17).” <em>Catalogue of Life</em>. Catalogue of Life. https://doi.org/10.48580/dft7. Elliott, Michael J., Jorrit H. Poelen, and José A. B. Fortes. 2023. “Signing Data Citations Enables Data Verification and Citation Persistence.” <em>Scientific Data</em> 10 (1). https://doi.org/10.1038/s41597-023-02230-y. Maniatis, Petros, Mema Roussopoulos, Thomas J Giuli, David SH Rosenthal, and Mary Baker. 2005. “The LOCKSS Peer-to-Peer Digital Preservation System.” <em>ACM Transactions on Computer Systems (TOCS)</em> 23 (1): 2–50. Poelen, Jorrit H. (ed.). 2023. “Nomer Corpus of Taxonomic Resources hash://sha256 /12051b8aa59930d6561a3ed46b7cf3f67a31a98445a457d78 894f6b8a8e81641 hash://md5/1ff6b3628d7afc15b882cc0c9b1c3815.” Zenodo. https://doi.org/10.5281/zenodo.8326175. Poelen, Jorrit, and José Augusto Salim. 2023. “Globalbioticinteractions/Nomer: 0.5.4.” Zenodo. https://doi.org/10.5281/zenodo.8329422.
分类名称对齐（Taxonomic name alignment）是整合生物多样性数据集以供复用过程中一项必要且通常耗时的任务。本出版物旨在通过重新打包生命名录（Catalogue of Life）中丰富的名称资源，实现快速（例如＞1000个名称/秒）、支持离线且可复现的名称对齐工作流。 <strong>引言</strong> 生命名录（Catalogue of Life，Bánki等人2023）是"……由专家汇编的全球物种名录汇编，旨在构建一份涵盖地球所有已知生物物种的综合名录。……"本数据出版物包含生命名录的可验证副本，以及NameUsage.tsv表的反向排序版本。本出版物的目标包括： 1. 为生命名录副本提供可签名的引用（Elliott、Poelen与Fortes 2023） 2. 筹备生命名录以纳入《分类学资源命名器语料库（Nomer Corpus of Taxonomic Resources，J. H. Poelen编辑，2023）》 3. 预处理生命名录，以支持使用Nomer（J. Poelen与Salim 2023）等工具实现优化索引与离线分类名称对齐。 总体而言，本出版物旨在借助生命名录提供的丰富信息，实现分类名称对齐，以助力名称列表与已知来源分类学资源之间实现快速、可复现且支持离线的对齐操作。本出版物支持的一项应用示例是通过https://github.com/globalbioticinteractions/name-alignment-template提供的分类名称对齐工具。该模板仓库通过GitHub Action实现自动化工作流，可将csv/tsv文件与达尔文核心归档（Darwin Core Archive）中的科学名称与常见分类学名称列表（如生命名录、NCBI分类法、综合分类学信息系统（Integrated Taxonomic Information System，ITIS）以及GBIF主干分类法）进行对齐。 <strong>方法</strong> 为获取并处理生命名录，我们执行了以下步骤： 1. 追踪并归档生命名录副本 2. 对NameUsage.tsv进行反向排序 3. 为处理后的资源分配别名 步骤1至3通过Preston（一款生物多样性数据追踪工具）进行记录与文档化。Preston不仅可以记录处理流程，还会包含所使用和生成的数字资源。 <strong>追踪与归档</strong> 为追踪并归档生命名录副本，执行以下命令： <pre><code>preston track https://download.catalogueoflife.org/col/latest_coldp.zip</code></pre> 执行该命令后，将下载https://download.catalogueoflife.org/col/latest_coldp.zip的副本并计算其sha256校验和（或哈希值），同时将下载过程记录为机器可读的RDF/N-Quads语句。本出版物中包含的副本的内容标识符（或sha256哈希值）可通过以下命令获取： <pre><code>preston alias --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd --remote https://zenodo.org/deposit/8327611/files https://download.catalogueoflife.org/col/latest_coldp.zip head -n1</code></pre> 其结果为： <pre><code><https://download.catalogueoflife.org/col/latest_coldp.zip> <http://purl.org/pav/hasVersion> <hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1> <urn:uuid:f00d82a2-d965-4b1d-9030-0a2f8833e004> .</code></pre> 这意味着<code>hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1</code>是编译本出版物时，由<code>https://download.catalogueoflife.org/col/latest_coldp.zip</code>生成的内容的内容标识符（或sha256哈希值）。 <strong>反向排序NameUsage.tsv</strong> 第二步是对追踪得到的生命名录数据包的部分内容进行预处理（或转换），并将其内容反向排序。我们使用以下Bash脚本完成该处理： <pre><code>(sed -u 1q; LC_ALL=C sort -r) | gzip</code></pre> 该脚本首先打印表头（即<code>sed -u 1q</code>），随后对后续内容进行反向排序（即<code>LC_ALL=C sort -r</code>），最后通过gzip压缩输出结果。本脚本属于本数据出版物的一部分，可通过以下方式获取： <pre><code>preston cat --remote https://zenodo.org/record/8327611/files/ hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd</code></pre> 为便于引用该脚本，我们创建了以下别名： <pre><code>preston alias urn:example:reverse-sort.sh hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd</code></pre> 完成反向排序脚本的编写并为其别名（即`urn:example:reverse-sort.sh`）完成文档化后，我们通过以下命令将该脚本应用于获取到的生命名录副本： <pre><code>preston cat --remote https://zenodo.org/record/8327611/files/ ‘zip:hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1!/NameUsage.tsv’ | preston bash --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd --remote https://zenodo.org/record/8327611/files/ -c urn:example:reverse-sort.sh</code></pre> 该处理过程得到的内容的sha256哈希值为<code>hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc</code>，相关信息记录于关联的Preston处理日志的第22行，可通过以下方式查看： <pre><code>preston cat --remote https://zenodo.org/record/8327611/files/ 'line:hash://sha256/8ba35deafc847f0d5d69d357241a431b7fd9b6f2735189575b2b7168d523caa9!/L22' </code></pre> <strong>为处理后的资源分配别名</strong> 我们定义了别名`col:NameUsage.tsv.gz`以简化对该结果的引用，命令如下： <pre><code>preston alias col:NameUsage.tsv.gz hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc</code></pre> 执行以下命令可列出生成的资源的前3行内容： <pre><code>preston cat --anchor hash://sha256/e7130fb557d9aee033ac7147f4d5c4c75f12223dd43e53c7cbb141372f9579cd --remote https://zenodo.org/record/8327611/files/ col:NameUsage.tsv.gz | gunzip | head -n3</code></pre> 其中<code>preston cat ...</code>用于打印生成的资源，<code>gunzip</code>用于解压结果，<code>head -n3</code>用于选取前3行。该操作的结果如下： <code>col:ID col:alternativeID col:nameAlternativeID col:sourceID col:parentID col:basionymID col:status col:scientificName col:authorship col:rank col:notho col:uninomial col:genericName col:infragenericEpithet col:specificEpithet col:infraspecificEpithet col:cultivarEpithet col:namePhrase col:nameReferenceID col:publishedInYear col:publishedInPage col:publishedInPageLink col:code col:nameStatus col:accordingToID col:accordingToPage col:accordingToPageLink col:referenceID col:scrutinizer col:scrutinizerID col:scrutinizerDate col:extinct col:temporalRangeStart col:temporalRangeEnd col:environment col:species col:section col:subgenus col:genus col:subtribe col:tribe col:subfamily col:family col:superfamily col:suborder col:order col:subclass col:class col:subphylum col:phylum col:kingdom col:sequenceIndex col:branchLength col:link col:nameRemarks col:remarks ffc77d7d-2ede-49ff-ab12-03410a1c25db 55434 93MTR 4WGYN 暂定接受 [Semiothisa] lapidata Warren, 1906 物种 动物学 合法 否 ff82a38f-348b-4fd7-891e-a2a68d8edce4 55434 93MTR 78YFQ 暂定接受 [Sabulodes] arnissa Druce, 1891 物种 动物学 合法 否</code> <strong>结果</strong> 如前文方法部分所述，本出版物生成了别名为`col:NameUsage.tsv.gz`、内容标识符为<code>hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc</code>的资源。该资源包含从https://download.catalogueoflife.org/col/latest_coldp.zip获取的生命名录数据包中NameUsage.tsv的反向排序副本，其内容标识符为<code>hash://sha256/d512d769a3e68b6f3be523b97f9f3c05b10d317607f95cb24ddcda27bded03e1</code>。 本数据出版物使用的处理工具包括：preston、bash、gzip、sed、head、sort。 <strong>讨论</strong> 本出版物旨在助力生命名录数据包在分类名称对齐工作流中的复用。尽管主要目标是为Nomer v0.4.5（J. Poelen与Salim 2023）生成可用资源，但也可设想其他应用场景，例如： 1. <strong>多副本保障数据安全（LOCKSS，Maniatis等人2005）</strong>：在生命名录基础设施之外留存生命名录数据包的完整副本 2. 演示如何使用Preston记录数据转换流程 3. 提供一份可流式读取的生命名录反向排序副本，可通过https://zenodo.org/record/8327611/files/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc获取，用于诸如查找首个包含<em>Enhydra lutris</em>（海獭）的记录这类工作流： <pre><code> curl -L 'https://zenodo.org/record/8327611/files/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc' | gunzip | grep "Enhydra lutris" | head -n1</code></pre> <strong>致谢</strong> 本研究立足于开源软件与开放获取数据集的贡献者们的工作，特此致谢！ <strong>参考文献</strong> 1. Bánki, O 等. 2023. 《生命名录名录（版本2023-08-17）》. <em>生命名录</em>. 生命名录. https://doi.org/10.48580/dft7. 2. Elliott, Michael J.、Jorrit H. Poelen与José A. B. Fortes. 2023. 《签名数据引用支持数据验证与引用持久性》. <em>Scientific Data</em> 10(1). https://doi.org/10.1038/s41597-023-02230-y. 3. Maniatis, Petros、Mema Roussopoulos、Thomas J Giuli、David SH Rosenthal与Mary Baker. 2005. 《LOCKSS对等网络数字保存系统》. <em>ACM Transactions on Computer Systems (TOCS)</em> 23(1): 2–50. 4. Poelen, Jorrit H.（编辑）. 2023. 《分类学资源命名器语料库 hash://sha256/12051b8aa59930d6561a3ed46b7cf3f67a31a98445a457d78894f6b8a8e81641 hash://md5/1ff6b3628d7afc15b882cc0c9b1c3815》. Zenodo. https://doi.org/10.5281/zenodo.8326175. 5. Poelen, Jorrit与José Augusto Salim. 2023. 《Globalbioticinteractions/Nomer: 0.5.4》. Zenodo. https://doi.org/10.5281/zenodo.8329422.
