SMILES transformer pretrained to predict 113 RDKit molecular descriptors, giving embeddings that carry physicochemical properties into ADMET models.
No providers recorded yet. Browse all providers
Predicting how a molecule will be absorbed, distributed, metabolized, excreted, and tolerated is one of the earliest filters in drug discovery, and it is chronically data-starved: the public ADMET benchmarks in Therapeutics Data Commons run from a few hundred to a few thousand labeled compounds. The standard remedy is to pretrain a language model on millions of unlabeled SMILES strings and reuse its embeddings downstream. Smile-to-Bert, from the B2SLab at the Universitat Politècnica de Catalunya with collaborators at the Technical University of Munich, changes what that pretraining task is.
Rather than recovering masked tokens, Smile-to-Bert regresses each SMILES string onto a 113-dimensional vector of RDKit-computed molecular descriptors — molecular weight, electrotopological state indices, ring counts, drug-likeness scores, and similar quantities. The premise is that fingerprint-and-descriptor models remain remarkably strong on ADMET leaderboards, so an embedding that already encodes those descriptors should carry information a purely structural encoder misses. Principal component projections of the learned embeddings show smooth gradients in exact molecular weight, QED drug-likeness, and fraction of sp3-hybridized carbons, confirming the descriptor signal survives into the representation.
This places Smile-to-Bert alongside chemical language models such as ChemBERTa and MoLFormer-XL, which pretrain by masked-token recovery on much larger SMILES corpora. The distinction is the supervision: Smile-to-Bert's embeddings are explicitly property-shaped, which turns out to make them complementary to — rather than a replacement for — structurally pretrained ones.
The encoder is a reduced BERT: four encoder layers, eight attention heads, an embedding size of 512, dropout 0.1, and a fixed sequence length of 100 tokens. Embeddings for non-padding tokens are averaged into a single 512-dimensional vector that feeds the descriptor head. Pretraining used 3,976,176 PubChem compounds, obtained by computing the 200 MapLight RDKit descriptors and discarding those with identical first and third quartiles plus the numerically unstable Ipc descriptor. Descriptors were median-and-IQR normalized and scaled to avoid vanishing gradients. Training ran 20 epochs with L1 loss, Adam with weight decay 0.01, separate learning rates of 2e-4 for the encoder and 5e-5 for the head, and 2,000 warm-up steps, distributed across four NVIDIA A40 GPUs.
Downstream evaluation covered all 22 TDC ADMET datasets under the package's scaffold splits and five prescribed seeds, attaching a two-layer feed-forward head and unfreezing only the last one or two encoder layers. Standalone, Smile-to-Bert takes the top result on one dataset — Spearman 0.363 on half-life — while a SMILES-to-IUPAC translation Transformer leads on nine and a two-encoder masked-token BERT on five. Combined models take the remaining eight, seven of them from the two-encoder pairing.
Smile-to-Bert is aimed at early-stage compound triage, where a medicinal chemistry team needs ADMET estimates for thousands of virtual candidates and has only small labeled sets to fit against. Its embeddings can be used directly as features for a lightweight classifier or regressor, or concatenated with those of another pretrained SMILES encoder to build a stronger ensemble — the configuration the authors recommend, and the one that helps most on metabolism, clearance, and toxicity endpoints. Because everything runs from SMILES text, no conformer generation or graph construction is required.
The study's most useful finding is a negative one honestly reported: pretraining directly on physicochemical descriptors yields embeddings that are too task-specific to beat broadly pretrained encoders on their own, yet carry enough orthogonal information to improve them in combination. Simple random forests over fingerprints and descriptors still top most TDC leaderboards, a gap the authors state plainly. The work is a preprint under a CC BY-ND license, and neither code repository carries a LICENSE file, which leaves reuse terms for the weights unstated. As a probe of what pretraining objective actually shapes a chemical embedding, it gives a clearer answer than most benchmark sweeps.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.