TabPFN Opens New Avenues for Small-Data Tabular Learning in Drug Discovery

Early-stage drug discovery often suffers from data scarcity
and
out-of-distribution (OOD) shifts, which constrain the reliability
of predictive models. While deep learning has advanced representation
learning from molecular and biological data, tabular modeling remains
indispensable, particularly in small-sample and OOD scenarios. For
more than a decade, gradient-boosted decision trees (GBDTs), such
as XGBoost, have been the dominant choice, yet their robustness is
limited under such conditions. TabPFN, a recently introduced transformer-based
tabular foundation model, enables accurate predictions on small data
sets without task-specific retraining. Applying TabPFN to a variety
of molecular data sets, we find that TabPFN performs on par with XGBoost
in classification but demonstrates clear and stable advantages in
regression, with its strongest gains on small and medium data sets
and under OOD evaluations. Feature and data ablations (10–90%)
further highlight its robustness, as performance degrades gracefully
and exhibits minimal sensitivity compared with tree ensembles. On
quantum tasks, TabPFN shows competitive accuracy on QM7 but is challenged
by the larger QM8 data set, where tree ensembles regain strength.
Beyond metrics, embedding analyses indicate smoother structure–property
relationships of TabPFN and enhanced class separability, reflecting
beneficial inductive biases rather than overfitting. Collectively,
these findings demonstrate that TabPFN offers a robust and data-efficient
alternative for tabular learning in drug discovery, shedding new light
on predictive modeling under small-data and OOD challenges.