MetaCoOc pre-formatted input files

MetaCoOc Pre-formatted Data Files This Zenodo record contains pre-formatted data files for use with MetaCoOc. MetaCoOc combines: community profiles derived from metagenomes - sample metadata, such as metadata submitted with NCBI SRA metagenome records biome annotations and curated sample classifications MetaCoOc currently supports taxonomic profiles generated by singleM and hosted in Sandpiper. Its Ingredients format can also represent other profiles with compatible taxonomy (Greengenes format) and sample information. Data releases and variants MetaCoOc data releases are versioned independently of both the MetaCoOc Python package and the Ingredients storage format. A complete published data-release identifier is: R<database_release>_<variant>_rev<revision> This record provides database release R226, snapshot revision 1, in two variants: R226_gtdb_rev1: Sandpiper profiles based on GTDB genomes. R226_globdb_rev1: Sandpiper profiles based on GlobDB genomes. The revision identifies a complete publication snapshot covering both variants, the SRA metadata and the standalone biome mapping. If any scientific input is corrected, all six files will be published under a new revision. Previous revisions remain immutable. The Ingredients archives are variant-specific. SRA metadata and the standalone biome mapping are shared by both variants. Files in this release ingredients_raw_R226_gtdb_rev1_format1.tar.gz ingredients_aggregated_R226_gtdb_rev1_format1.tar.gz ingredients_raw_R226_globdb_rev1_format1.tar.gz ingredients_aggregated_R226_globdb_rev1_format1.tar.gz sra_metadata_R226_rev1.tsv.gz sample_to_biome_R226_rev1.tsv.gz Ingredients archives The main files in this record are pre-formatted Ingredients archives used by MetaCoOc. The community profiles were generated by SingleM and downloaded from the Sandpiper Zenodo repository. MetaCoOc was used to format these profiles into Ingredients objects. For each supported Sandpiper database variant, both raw and aggregated Ingredients are provided: ingredients_raw_<database_release>_<variant>_rev<revision>_format<format_version>.tar.gz ingredients_aggregated_<database_release>_<variant>_rev<revision>_format<format_version>.tar.gz These archives extract to Ingredients directories. Each directory contains the sparse taxon-by-sample matrices, sample and taxon labels, biome annotations, and a manifest. The manifest records the data release, Ingredients schema version, generation date, matrix details, and component filenames. The two Ingredients forms differ as follows: Raw: coverage is retained at the taxonomic rank reported by SingleM and does not include coverage assigned to descendant taxa. Aggregated: coverage is propagated through the taxonomy so that higher-rank taxa include coverage from their descendants. SRA metadata NCBI SRA metadata for the metagenomes screened in each Sandpiper release are included here. This metadata can be parsed for use in MetaCoOc searches and filtering. The parsing workflow was adapted from public_sequencing_metadata_corrections. Further details are described in the MetaCoOc documentation. Because this table can be large and is not required for the default taxonomic or biome-based workflows, the MetaCoOc downloader does not fetch it by default. Request it explicitly with `--include-metadata`. Downloading with MetaCoOc To download the raw and aggregated Ingredients for one variant: metacooc download --data-release R226_gtdb_rev1 or: metacooc download --data-release R226_globdb_rev1 To also download the shared SRA metadata table: metacooc download --data-release R226_gtdb_rev1 --include-metadata The selected command downloads both Ingredients archives for that variant. The SRA metadata table is downloaded only when `--include-metadata` is supplied. The standalone sample_to_biome file is not downloaded automatically because the same biome mapping is already included in each Ingredients archive. Biome Annotations Two-level Biome annotations were assigned through a combination of manual curation and automated retrieval of metadata for each SRA metagenome. Metagenomes with clear and detailed metadata could be relatively simply assigned to the broad biome classes we used. Manual curation, often including non-standard entry fields, was required for a significant number of metagenomes. If a classification was not possible based on metadata, biomes were classified as unspecified or unclassified. Biome annotations are already embedded in each Ingredients directory as sample_to_biome.tsv. The separately downloadable sample_to_biome_R226_rev1.tsv.gz is provided for inspection, and reuse outside the pre-formatted Ingredients. Metadata (sra_metadata_<database_release>_rev<revision>.tsv.gz) and Biome annotations are version shared. Biome annotations are in the sample_to_biome.tsv file in the Ingredients archives, but are also exposed in this repo for ease of access as sample_to_biome_<database_release>_rev<revision>.tsv.gz. Versioning The identifier R226_gtdb_rev1 describes: database release R226 database variant gtdb publication snapshot revision 1. It does not describe the Ingredients storage schema. Ingredients directories contain a separate integer format_version in their manifests. The archives in this release use Ingredients format version 1, represented by _format1 in their filenames. Changelog R226 revision 1 — Ingredients format 1 Replaced semantic data versions with database release, variant and revision identifiers: R226_gtdb_rev1 and R226_globdb_rev1 Raw and aggregated Ingredients for the R226 GTDB- and GlobDB-based profiles. Introduced global snapshot revisions covering both Ingredients variants and the shared support files. Added directory-based Ingredients storage with manifests and lazy loading of large matrices. Versioned the Ingredients storage schema independently from the data release - format_version = 1 Made the large SRA metadata table an optional MetaCoOc download. The below versions are no longer compatible with MetaCoOc: version 1.1.0 Built from Sandpiper-1.1.0 profiles Ingredients class updated: matrices transposed to be in line with commonly used orientation of taxa x samples; class updated to more efficiently filter, search and count. globdb ingredients introduced Biome distributions updated after more curation version 1.0.1 Underlying architecture of the Ingredients class was updated and necessitates a fresh upload. Biome distribution data can also be accessed from the Ingredients class now version 1.0.0 In-line with the update and release of Sandpiper1.0.0 - files in this repository were updated and versioning introduced