Pretrained Transformer Encoder for SMILES Strings

Pretrained Transformer Encoder Parameters

This component provides the pretrained parameters for a transformer encoder designed to extract feature representations from SMILES strings. The model was trained using masked token prediction to capture intricate patterns and long-range dependencies within molecular sequences.

The transformer architecture includes:

10 sequential transformer blocks,

Multi-head self-attention for contextualized token embeddings,

Position-wise feed-forward layers with Gaussian Error Linear Unit (GELU) activation, residual connections, and layer normalization.

The global molecular representation is derived from the start token embedding, which aggregates sequence-wide information during self-attention computations.

Pretraining Dataset

This component provides the dataset used to pretrain the transformer encoder. It integrates SMILES strings from the following sources:

ChEMBL 33: ~2.4 million bioactive molecules with drug-like properties,

GuacaMol v1: ~1.6 million molecules derived from ChEMBL 24,

MOSES: ~1.8 million molecules selected from ZINC 15 for diversity and medicinal chemistry suitability,

BindingDB: ~1.2 million unique small molecules bound to proteins,

PDBbind v2020: ~15,710 unique small molecules bound to proteins.

This model has been optimized for drug discovery applications, including protein-ligand binding affinity prediction, and can serve as a foundational tool for researchers working on cheminformatics, computational biology, and medicinal chemistry.