Probabilistic representation learning of baseline transcriptomes enables mechanistic interpretation of anticancer drug responses

Anticancer drug responses vary widely across cancer cell lines, reflecting intrinsic transcriptional states. These baseline states shape cellular vulnerability prior to treatment. While machine learning models can predict drug sensitivity from high-dimensional gene expression features, they provide limited insight into shared and drug-specific mechanisms of action. We present a probabilistic representation learning framework that embeds baseline transcriptomes into a low-dimensional latent space using variational autoencoders and leverages these latent factors to dissect common and compound-selective determinants of drug sensitivity. Using baseline gene expression and IC50 values for five mechanistically diverse anticancer agents from a public cancer cell line panel, we show that latent representations combined with multi-head ridge regression improve prediction performance compared with gene-level models while yielding more stable behavior than alternative autoencoder and contrastive learning approaches. Latent factors with high predictive importance are decomposed into shared axes, which are consistently associated with response to all reference drugs, and drug-specific axes, which show selective importance for individual compounds. Pathway enrichment analysis reveals that shared axes capture canonical cancer-related programs including adhesion, extracellular matrix organization, and MAPK and PI3K-Akt signaling, whereas drug-specific axes highlight more restricted, pathway-level signatures. Applying the framework to the novel anticancer compound QAL333 demonstrates that its baseline sensitivity is governed primarily by shared vulnerability axes, and latent profiles of SW620 and MDA-MB-231 cells align with differential in vivo responses in zebrafish xenograft models. These results support the biological relevance of the learned latent representations.