Benchmarking of modification-aware basecalling models

Nanopore direct RNA sequencing (DRS) holds promise for advancing our understanding of the epitranscriptome by enabling the direct detection of RNA modifications in native RNA molecules. Recently, Oxford Nanopore Technologies (ONT) has released basecalling models capable of detecting several RNA modification types at single-molecule level. However, their accuracy, sensitivity and specificity, as well as potential cross-reactivity against other modification types, remains largely unexplored. Here, we systematically benchmark modification-aware models by evaluating their performance on a highly-multiplexed panel of synthetic molecules with known RNA modification types and varying modification frequencies covering all possible sequence contexts, as well as on biological rRNA from E. coli and S. cerevisiae wild type and knockout strains deficient in specific RNA modifications. We find that modification-aware models reliably detect diverse RNA modification types across a broad range of sequence contexts; however, they are prone to elevated false positive rates and exhibit notable cross-reactivity with other RNA modification types. We show that the use of modification-free controls allows significant –yet incomplete– removal of false positives, thus constituting an essential control that should be included when using these models. Finally, for those modifications for which modification-aware models are not available, we demonstrate that basecalling 'error' patterns and alterations in current features can identify differentially modified sites. Overall, our results underscore the utility and accuracy of modification-aware basecalling models for RNA modification detection, while also highlighting the importance of including diverse control samples to mitigate false positive rates.