When Do Models Win? A Learning Curve Benchmark for Molecular Property Prediction in Low-Data Regimes — Test-Set Predictions

Per-molecule test-set predictions underlying the learning-curve benchmark reported in "When Do Models Win? A Learning Curve Benchmark for Molecular Property Prediction in Low-Data Regimes" (Journal of Cheminformatics, under review). This deposit contains five files. One is the data itself; the other four exist so that the data can be verified, read, and understood without guesswork. jcheminf_predictions_v1.tar.gz (231.85 MB) The data. Unpacking produces a single directory, jcheminf_predictions_v1/, holding 4,764 entries: 4,761 prediction files plus copies of the README, the manifest, and the loader, so that the archive is self-describing even when downloaded on its own. Uncompressed it occupies 243.1 MB. The prediction files are split across four dataset directories: QM9 (2,557 files), ESOL (720), Lipophilicity (774) and BACE (710). Each .npz file corresponds to one (model, seed, target, training size) cell and holds four arrays. test_preds is the model's prediction for each held-out molecule and test_true is the ground truth in the same order; both are float32 in z-score space. y_mean and y_std are the mean and standard deviation of that cell's training split, and are what recover physical units: prediction * y_std + y_mean returns the original scale, and an RMSE computed in z-space need only be multiplied by y_std. Three points require care. First, the QM9 homo, lumo and gap targets are stored in Hartree; multiply by 27.2114 to obtain the eV values reported in the manuscript. Second, y_std is a scalar in most files but a 12-element array in 92 of them, because the QM9 multi-task models predict twelve properties jointly; the manifest's y_std_ndim column distinguishes the two cases. Third, the number of held-out molecules can differ between models within the same cell — on QM9, 2,281 files cover 9,997 molecules while 276 cover 8,777, because the 3D models drop molecules for which ETKDG conformer generation failed. Ensembles must therefore be formed over the majority test set only. The bundled loader handles this automatically. CHECKSUM.txt (97 bytes) A single line carrying the SHA-256 digest of the archive, e759ae4ba11dac297bfc5dad10a2a7450e8da84c35dc8fc9500980f61ff0476f. Running "sha256sum -c CHECKSUM.txt" confirms that a 232 MB download arrived intact. manifest.csv (967 KB, 4,761 rows) An index of the archive's contents that can be read without unpacking it, so the composition of the deposit can be inspected before committing to the full download. It carries fourteen columns. dataset, dataset_dir, file (the path inside the archive), model, variant, seed, target and train_size identify what each file is. n_test, y_mean, y_std and y_std_ndim make unit recovery and test-set size available without opening any .npz. bytes and sha256 are per-file checksums, so an individual file can be verified rather than only the archive as a whole. The variant column reads na for 4,500 files and 6 for 261. The latter is the fusion depth of GTCA-Cat, the model that Section 3.5 of the manuscript excludes from the ensemble analysis; the loader filters it out by default. load_predictions.py (6.7 KB) A reader that requires nothing beyond NumPy. Run with no arguments it prints a summary of the archive; given a dataset and a training size — "python load_predictions.py esol 100" — it prints per-model RMSE and the ensemble values for that cell. --target selects a QM9 target and --seed restricts the output to a single seed. Its default behaviour reproduces the manuscript exactly: GTCA is excluded, only the majority test set is used, and QM9 targets are converted to eV. Running ESOL at N=100 therefore yields the 17 models and ensemble RMSE of 1.6027749 reported in the paper. Pass --all-variants to include everything. This is the file that makes the deposit useful. It was used to recompute the published ensemble CSVs from scratch, reproducing them to within a relative deviation of 1.5e-07 — evidence that the archive alone is sufficient to reproduce the paper's ensemble results independently. README.md (8.6 KB, 10 sections) The documentation. Section 1 states the scope: which datasets, training sizes and models are present, and what is deliberately absent. Section 2 records the verification performed, with the ensemble-reproduction results and worst-case deviations for all four datasets. Sections 3 and 4 cover the directory layout and file-naming scheme, Section 5 the file format and unit recovery described above, Section 6 the per-dataset test-set sizes, Section 7 the model list and family assignments, Section 8 the manifest columns, Section 9 usage, and Section 10 the licence (CC BY 4.0). Section 8 documents one result worth flagging in advance. The manifest contains 615 pairs of files sharing an identical SHA-256. This is not a packaging error: on QM9 the scaffold split is invariant to the random seed, so certain models produce byte-identical predictions across seeds. The mechanism is analysed in the manuscript's Discussion under "Seed-invariant scaffold splits", and the README points there. Coverage is restricted to the training sizes at which every model has a saved prediction, matching the ensemble analysis reported in Section 4.6 of the manuscript: QM9 N = 50, 100, 200, 500, 1,000, 3,000; ESOL N = 50, 100, 200, 375; Lipophilicity N = 50, 100, 200, 500, 1,000; BACE N = 50, 100, 200, 500. Analysis code is available at https://github.com/Jun4M/WhenDoModelsWin and archived at doi:10.5281/zenodo.19562604.