Foundation model representation-geometry analysis on a genomic testbed: 10-condition factorial dataset, 4-metric robustness analysis, and trained checkpoints

Reproducibility package for the manuscript 'When do foundation-model representations transcend trivial baselines? Objective design governs representation geometry beyond scale' (Tanigawa & Iwaki, 2026, submitted). Question. Foundation models are assumed to learn representations that transcend trivial statistical baselines of their input domain — k-mer composition in genomics, n-gram statistics in NLP, colour histograms in vision — but whether they actually do is rarely tested rigorously. Genomics offers a rare configuration in which a trivial baseline (k-mer composition), a ground-truth distance (evolutionary divergence times curated by literature consensus), and a residual statistic (partial Mantel) are all precisely defined. We exploit this configuration as a clean testbed for foundation-model representation evaluation, using splice sites as a controlled substrate. Contents: - 52,280 introns (104,560 donor and acceptor sequences) from 140 species (39 Mammalia, 80 Insecta, 21 Nematoda) across 5 functional gene categories - 10-condition factorial training-objective dataset on HyenaDNA-medium-160k (6.55 M parameters): factorial axes are loss (next-token / contrastive / classification / regression / adversarial), label resolution (phylum / species / category), label correctness, and initialisation - Fine-tuned model checkpoints (9 single-seed final plus 24 multi-seed) and the full n=20 per-seed Mantel results (180 files) reproducing Table S3 and Figure 3 - 34 model embeddings (HDF5) on the held-out aging + DNA-repair evaluation set (45,080 sequences) - Residual Mantel test results (9,999 permutations, BH-FDR per phylum) for all conditions - Five-model GFM benchmark embeddings (Evo2 7B, NT-v2 500M, DNABERT-2 117M, DNABERT-S 117M, HyenaDNA-large 243M) - Four-metric representation-alignment robustness analysis: residual Mantel, representational similarity analysis (RSA Spearman), k-nearest-neighbour preservation @ k=5, k-NN phylogenetic-retrieval mean average precision @ k=5 - End-to-end reproducible pipeline: the original numbered scripts plus the revision analyses (n=20 extended-seed campaign, matched-adaptation control, DNABERT-2 multi-architecture replication, ESM-2 protein-portability test, and confound controls) - 5 supplementary tables: gene lists, per-species N content, full n=20 per-seed Mantel results, 5-model benchmark, per-seed 4-metric robustness Headline results: - Across five GFMs (6.55 M–7 B parameters), larger-scale zero-shot pretrained models do not reliably recover residual evolutionary geometry under this evaluation; only DNABERT-S (the contrastively-pretrained 117 M model) captures any beyond-composition signal. - Training-objective design, not model scale, is the decisive lever: a matched-adaptation control (an identical lightweight head on frozen embeddings) gives the 7 B model no advantage over a 6.55 M one, while a directed species-aware objective on the small network recovers beyond-baseline geometry that zero-shot scale does not. - The species-fine pretrained 'default' recipe is the worst Tier-1 design; four single-substitution alternatives (phylum-coarse contrastive, supervised species classification, distance regression, random-init species contrastive) all outperform it (paired Δ ≥ +0.058, p < 0.001, n = 20 seeds). - Within Tier 1, objectives optimise different geometry scales: phylum-coarse contrastive supervision preferentially improves global interspecies alignment (paired Δ = +0.073, Wilcoxon p = 1.9e-6, n = 20) while species-fine supervision better preserves local neighbourhoods: a global-vs-local geometry trade-off that replicates on a Transformer backbone (DNABERT-2) and transfers to protein language models (ESM-2). Reproducibility: the n=20 verification (180 trainings) reproduces on a single NVIDIA RTX 3090 in approximately 27 hours wall-clock (the core single-seed pipeline runs in ~5 hours). All scripts are MIT-licensed; data and embeddings are CC BY 4.0. Related manuscript: Tanigawa & Iwaki (2026), under review.