Indexing and searching petabase-scale nucleotide resources

Searching vast and rapidly growing nucleotide content in resources, such as runs in Sequence Read Archive and assemblies for whole genome shotgun sequencing projects in GenBank, is currently impractical for most researchers. We present Pebblescout, a tool that navigates such content by providing indexing and search capabilities. Indexing uses dense sampling of the sequences in the resource. Search finds subjects (runs or assemblies) that have short sequence matches to a user query, with well-defined guarantees, and ranks them using informativeness of the matches. Eight databases that index over 3.7 petabases were created to illustrate the functionality of Pebblescout. They can be interactively searched at https://pebblescout.ncbi.nlm.nih.gov. Here we show that for a wide range of query lengths, Pebblescout provides a data-driven way for finding relevant subsets of large nucleotide resources; reducing the resources needed for downstream analysis significantly. We compare Pebblescout to MetaGraph and Sourmash and show that both methods do not report some results correctly reported by Pebblescout.