Foundation model representation-geometry analysis on a genomic testbed: 10-condition factorial dataset, 4-metric robustness analysis, and trained checkpoints
收藏资源简介：
<p>Reproducibility package for the manuscript <em>"When do foundation model representations transcend trivial baselines? Training-objective design as the decisive factor"</em> (Tanigawa & Iwaki, 2026, submitted).</p> <p><strong>Question.</strong> Foundation models are assumed to learn representations that transcend trivial statistical baselines of their input domain — <em>k</em>-mer composition in genomics, <em>n</em>-gram statistics in NLP, colour histograms in vision — but whether they actually do is rarely tested rigorously. Genomics offers a rare configuration in which a trivial baseline (<em>k</em>-mer composition), a ground-truth distance (evolutionary divergence times curated by literature consensus), and a residual statistic (partial Mantel) are all precisely defined. We exploit this configuration as a clean testbed for foundation-model representation evaluation, using splice sites as a controlled substrate.</p> <p><strong>Contents:</strong></p><ul><li>52,280 splice-site sequences from 140 species (39 Mammalia, 80 Insecta, 21 Nematoda) across 5 functional gene categories</li><li>10-condition factorial training-objective dataset on HyenaDNA-medium-160k (6.55 M parameters): factorial axes are loss (next-token / contrastive / classification / regression / adversarial), label resolution (phylum / species / category), label correctness, and initialisation</li><li>30 multi-seed fine-tuned model checkpoints (6 conditions × 5 seeds)</li><li>34 model embeddings (HDF5) on the held-out aging + DNA-repair evaluation set (45,080 sequences)</li><li>Residual Mantel test results (9,999 permutations, BH-FDR per phylum) for all conditions</li><li>Five-model GFM benchmark embeddings (Evo2 7B, NT-v2 500M, DNABERT-2 117M, DNABERT-S 117M, HyenaDNA-large 243M)</li><li>Four-metric representation-alignment robustness analysis: residual Mantel, representational similarity analysis (RSA Spearman), k-nearest-neighbour preservation @ k=5, k-NN phylogenetic-retrieval mean average precision @ k=5</li><li>22 Python scripts forming the end-to-end reproducible pipeline (~5 hours wall-clock on a single RTX 3090)</li><li>5 supplementary tables: gene lists, per-species N content, full 30-seed Mantel results, 5-model benchmark, per-seed 4-metric robustness</li></ul> <p><strong>Headline results:</strong></p><ul><li>Across five GFMs (6.55 M–7 B parameters), larger-scale zero-shot pretrained models do not reliably recover residual evolutionary geometry under this evaluation; only DNABERT-S (the contrastively-pretrained 117 M model) captures any beyond-composition signal.</li><li>Random-label contrastive perturbation alone recovers ~80% of the lift achievable with real species labels, indicating that bias-erasure of pretraining priors dominates label-derived content.</li><li>The species-fine pretrained "default" recipe is the worst Tier-1 design; four single-substitution alternatives (phylum-coarse contrastive, supervised species classification, distance regression, random-init species contrastive) all outperform it (paired Δ ≥ +0.045, p ≤ 0.013, n = 5 seeds).</li><li>Within Tier 1, objectives optimise different geometry scales: phylum-coarse contrastive supervision preferentially improves global interspecies alignment (paired Δ = +0.066, p = 0.0001) while species-fine supervision better preserves local neighbourhoods — a global-vs-local geometry trade-off, robust across four representation-alignment assays.</li></ul> <p><strong>Reproducibility:</strong> the entire 30-run multi-seed verification can be reproduced on a single NVIDIA RTX 3090 in approximately 5 hours wall-clock. All scripts use deposit-relative paths and are released under the deposit's license; data and embeddings are CC BY 4.0.</p> <p><strong>Related manuscript:</strong> Tanigawa & Iwaki (2026), submitted to Nature Machine Intelligence.</p>
<p>本论文的可复现性配套资源：《基础模型（foundation model）表征何时能超越平凡基线？训练目标设计作为决定性因素》（Tanigawa与Iwaki，2026年，已投稿）。</p> <p><strong>问题提出</strong>：学界通常假设基础模型能够学习到超越其输入域平凡统计基线的表征——例如基因组学中的k聚体（k-mer）组成、自然语言处理中的n元语法（n-gram）统计、计算机视觉中的颜色直方图（colour histograms），但鲜少有研究对这一假设进行严格验证。基因组学领域存在一种罕见的理想实验配置：其平凡基线（k聚体组成）、经文献共识整理的真实进化距离（进化分歧时间）以及残差统计量（偏曼特尔检验（partial Mantel test））均可被精确定义。我们利用这一配置作为基础模型表征评估的干净测试平台，并以剪接位点（splice sites）作为受控实验基底。</p> <p><strong>数据集内容</strong>:</p><ul><li>140个物种的52280条剪接位点序列（其中哺乳纲39种、昆虫纲80种、线虫纲21种），涵盖5个功能基因类别</li><li>基于HyenaDNA-medium-160k（655万参数）的10条件析因训练目标数据集，析因维度包括：损失函数类型（下一词预测、对比学习、分类、回归、对抗学习）、标签分辨率（门纲目、物种、功能类别）、标签正确性以及初始化方式</li><li>30个多种子微调的模型检查点（6种训练条件 × 5个随机种子）</li><li>34个模型嵌入文件（HDF5格式），取自预留的衰老与DNA修复评估集（共45080条序列）</li><li>所有训练条件下的残差曼特尔检验结果（9999次置换，按门纲目进行BH-FDR假发现率校正）</li><li>5个基础模型（General Foundation Model, GFM）基准嵌入：Evo2 7B、NT-v2 500M、DNABERT-2 117M、DNABERT-S 117M、HyenaDNA-large 243M</li><li>4项表征对齐鲁棒性分析指标：残差曼特尔检验、表征相似性分析（RSA Spearman相关）、k=5时的k近邻保留率、k=5时的k近邻系统发育检索平均精度均值</li><li>22个Python脚本构成的端到端可复现实验流水线（单张RTX 3090上运行耗时约5小时）</li><li>5份补充表格：基因列表、各物种的N碱基占比、全部30个随机种子的曼特尔检验完整结果、5个模型的基准测试结果、每个随机种子的4指标鲁棒性分析结果</li></ul> <p><strong>核心结果</strong>:</p><ul><li>在5个基础模型（参数规模655万至70亿）中，大规模零样本预训练模型无法在该评估中可靠还原残差进化几何结构；仅DNABERT-S（经过对比预训练的117M参数模型）能够捕捉到超越k聚体组成的信号。</li><li>仅使用随机标签的对比学习扰动即可还原约80%的真实物种标签所能带来的性能提升，这表明预训练先验的偏差消除主导了标签衍生的表征内容。</li><li>以物种为粒度进行预训练的“默认”训练方案是表现最差的一级（Tier 1）设计；四种单一替换的替代方案（门纲目粗粒度对比学习、有监督物种分类、距离回归、随机初始化的物种对比学习）均优于该默认方案（配对差值Δ≥+0.045，p≤0.013，n=5个随机种子）。</li><li>在一级训练设计中，不同优化目标适配不同的几何结构尺度：门纲目粗粒度对比监督优先提升全局跨物种对齐效果（配对差值Δ=+0.066，p=0.0001），而物种细粒度监督则更能保留局部邻域结构——这一全局-局部几何结构权衡在4种表征对齐分析中均表现稳健。</li></ul> <p><strong>可复现性说明</strong>：整个30次多种子验证实验可在单张NVIDIA RTX 3090上于约5小时内完成复现。所有脚本均使用相对于本配套资源的相对路径，并基于本数据集包的许可协议发布；数据与嵌入文件采用CC BY 4.0许可协议。</p> <p><strong>相关论文</strong>：Tanigawa与Iwaki（2026），已投稿至《Nature Machine Intelligence》。</p>
