Gene-language models are whole genome representation learners

The language of genetic code embodies a complex grammar and rich syntax of
interacting molecular elements. Recent advances in self-supervision and
feature learning suggest that statistical learning techniques can identify
high-quality quantitative representations from inherent semantic
structure. We present a gene-based language model that generates
whole-genome vector representations from a population of 16
disease-causing bacterial species by leveraging natural contrastive
characteristics between individuals. To achieve this, we developed a
set-based learning objective, AB learning, that compares the annotated
gene content of two population subsets for use in optimization. Using this
foundational objective, we trained a Transformer model to backpropagate
information into dense genome vector representations. The resulting
bacterial representations, or embeddings, captured important population
structure characteristics, like delineations across serotypes and host
specificity preferences. Their vector quantities encoded the relevant
functional information necessary to achieve state-of-the-art genomic
supervised prediction accuracy in 11 out of 12 antibiotic resistance
phenotypes.