KERMT

An ADME dataset is small: the public Biogen set is 3,521 molecules across six endpoints, and even a well-instrumented internal program measures intrinsic clearance or permeability on a few thousand compounds. A graph network trained from scratch on that much data learns the assay rather than the chemistry underneath it. Pretraining on unlabelled molecules is the standard remedy, and the recipe for molecular graphs bolts several chemistry-specific self-supervised tasks onto one encoder — predict an atom's context, a bond's context, which functional groups are present — each contributing a loss term whose weight is tuned by hand.

KERMT (Kinetic GROVER Multi-Task) is NVIDIA's graph-transformer take on that recipe, developed with Merck & Co. as an enhanced reimplementation of the GROVER architecture from Tencent AI Lab. The engineering is the point of the first release: distributed pretraining through PyTorch DDP, GPU-accelerated atom and bond featurization through cuik-molmaker, automated hyperparameter search, and fine-tuning that trains correlated ADMET endpoints jointly instead of one model per assay. That work found the multi-task gain grew with dataset size rather than shrinking, the opposite of the usual expectation that pretraining pays off mainly in the low-data limit, and published two multi-task ADMET splits for future benchmarking.

The current release, Contrastive KERMT (v2.0), rebuilds the pretraining objective. Rather than treating reconstruction and contrast as auxiliary regularisers with separately tuned weights, it expresses SMILES reconstruction from the graph-derived latent, in-batch contrastive discrimination (cMIM) and the chemistry vocabulary tasks as unit-weighted log-probability factors in one probabilistic latent-variable objective. Nothing is left to weight.

#Key Features

• Graph in, SMILES out: A transformer decoder reconstructs the canonical SMILES string from the graph-derived latent code, forcing that latent to carry the full structure rather than only whatever the context-prediction tasks happen to reward.
• One objective, no loss weights: Reconstruction, contrastive discrimination and chemistry self-supervision enter as unit-weighted factors of one probabilistic objective, removing the weight-tuning sweep multi-objective molecular pretraining normally requires.
• Multi-task readout with per-task heads: Fine-tuning attaches task-specific MLP heads to a shared backbone, keeping representation learning shared while limiting negative transfer between endpoints whose chemistry pulls in different directions.
• Frozen embeddings or fine-tuning: The repository ships embedding extraction that runs the released checkpoint without gradients, emitting four 800-dimensional readouts plus the 512-dimensional projected latent. Headline ADMET accuracy comes from a fine-tuned head, not from zero-shot prediction.

#Technical Details

The encoder combines local message passing with global self-attention: hidden size 800, six message-passing-plus-attention layers, four attention heads per layer, one multi-task block, PReLU activations and dropout 0.1. The v2.0 decoder is three transformer layers with eight heads, a 512 hidden and latent dimension, 2048 feed-forward width and rotary positional encoding, giving a released checkpoint of 7.06 x 10^7 parameters. Input is a canonical SMILES string that RDKit featurizes into a 2D atom-and-bond graph. Pretraining ran 100 epochs over roughly 11.1 million deduplicated canonical SMILES — an 11M ZINC15 and ChEMBL base pool following the GROVER protocol, augmented with the Biogen ADMET (3.5K), ExpansionRX (7.6K) and ChEMBL-MT (114K) molecule sets — split by Bemis-Murcko scaffold. Downstream evaluation fine-tunes the checkpoint on those three benchmarks and reports MAE with Pearson and Spearman correlation per endpoint, under scaffold, temporal and Taylor-Butina cluster splits respectively. Averaged over significantly improved endpoints, the contrastive pretraining beats the KERMT baseline by 7.6% on Biogen, 9.9% on ExpansionRX and 9.5% on ChEMBL-MT, and adding those ADME-adjacent molecules to the pretraining corpus improves transfer further.

#Applications

The target user is a computational chemist with internal assay data for a handful of ADME endpoints — microsomal clearance, Caco-2 or MDR1-MDCK permeability, kinetic solubility, LogD, plasma protein binding, hERG inhibition — who wants predictions good enough to triage designs before synthesis. The workflow is to fine-tune the pretrained backbone on those endpoints jointly, then score virtual libraries; the same checkpoint doubles as a fixed featurizer for similarity search, clustering and latent-space visualization when labels are absent. The accelerated implementation targets industrial scale, where featurization and hyperparameter search dominate wall-clock time.

#Impact

KERMT is one of the few chemical pretrained models validated inside a pharmaceutical program: the first study reports results on Merck internal datasets alongside public ones, and its two published multi-task ADMET splits give the field a shared basis for comparing fine-tuning methods that had been evaluated on incompatible splits. Distribution is split: source code is Apache-2.0, while the v2.0 weights carry the NVIDIA Open Model License, a custom non-OSI license permitting commercial use. The checkpoint is downloadable from NGC and Hugging Face, with no hosted inference endpoint. Three limits are worth stating: the model reads 2D molecular graphs, so conformer-dependent properties lie outside what the representation encodes; every reported ADMET number comes from fine-tuning the backbone on labelled data; and both descriptions of the model remain preprints without peer review.