SPECTRA

Data for SPECTRA repository, associated with the publication: "Evaluating generalizability of artificial intelligence models for
molecular datasets"

SPECTRA or the spectral framework for comprehensive model evaluation is a novel way to evaluate, compare, and understand model generalizability. For a given model and input data, SPECTRA plots model performance as a function of decreasing cross-split overlap and reports the area under this curve as a measure of generalizability. We apply SPECTRA to 18 sequencing datasets with associated phenotypes ranging from antibiotic resistance in tuberculosis to protein-ligand binding to evaluate the generalizability of 19 state-of-the-art deep learning models, including large language models, graph neural networks, diffusion models, and convolutional neural networks. We show that existing splits provide an incomplete assessment of model generalizability. With SPECTRA, we find as cross-split overlap decreases, deep learning models consistently exhibit a reduction in performance in a task- and model-dependent manner. Although no model consistently achieved the highest performance across all tasks, we show that deep learning models can generalize to previously unseen sequences on specific tasks. SPECTRA paves the way toward a better understanding of how foundation models generalize in biology. This repository contains data needed to run SPECTRA analysis in the GitHub associated with this project: https://github.com/mims-harvard/SPECTRA.