GROVER

Masking an atom in a molecular graph and asking a network to name it is a far easier question than masking a word in a sentence. English has tens of thousands of word types; a drug-like molecule is built from roughly ten atom types, and guessing "carbon" answers most of the masked positions. The self-supervised objective that made BERT work therefore carries very little signal when transplanted onto chemistry.

GROVER (Graph Representation frOm self-superVised mEssage passing tRansformer) replaces that target with a contextual property: a statistical key describing the local subgraph around the masked atom or bond, written as a string such as C_N-DOUBLE1_O-SINGLE1 for a carbon with one double-bonded nitrogen and one single-bonded oxygen within radius one. Extracted over the pretraining corpus, these keys yield 2,518 distinct node labels and 2,686 edge labels — a vocabulary large enough to force the encoder to represent real local chemistry. A second, graph-level task predicts which of 85 RDKit-derived functional-group motifs a molecule contains, so supervision at both scales comes from pattern matching rather than from wet-lab measurements, and no labelled property data enters pretraining at all.

GROVER was built at Tencent AI Lab with the Beijing National Research Center for Information Science and Technology at Tsinghua University, and presented at NeurIPS 2020. Its name is shared with an unrelated DNA language model from another group; this entry covers the molecular graph encoder only.

#Key Features

• Contextual property prediction: Node- and edge-level masking predicts a context-aware subgraph key instead of an atom or bond type, removing the label ambiguity that makes naive atom masking nearly trivial.
• Motif prediction from domain knowledge: A multi-label graph-level head predicts the presence of 85 RDKit functional groups, giving whole-molecule supervision without any property labels.
• GTransformer backbone: Message-passing networks produce the queries, keys and values, so local subgraph structure enters the attention block and the transformer layer then relates every node to every other.
• Dynamic message passing: The hop count per layer is redrawn from a uniform or truncated-normal distribution each epoch, randomising each node's receptive field and generalising better than a fixed hop count.
• Fingerprint mode as a shipped function: main.py fingerprint emits atom-pooled, bond-pooled or concatenated embeddings from the pretrained checkpoint with no training step, so the encoder can be used as a frozen descriptor source.
• Long-range residual connection: A single skip from the input features to the final layers replaces the transformer's short-range residuals, easing the over-smoothing of deep message passing.

#Technical Details

Two sizes were pretrained and released: GROVER base at roughly 48M parameters and GROVER large at roughly 100M, differing only in hidden size. Pretraining used 11 million unlabelled molecules sampled from ZINC15 and ChEMBL, 10% held out for model selection, on 250 Nvidia V100 GPUs for 2.5 days (base) and 4 days (large). Fifteen percent of node and edge labels are masked per graph, at a context radius of one.

Evaluation covered 11 MoleculeNet benchmarks spanning physiology, biophysics, physical chemistry and quantum mechanics, under scaffold splitting at 8:1:1 over three seeds. GROVER large improved on the prior best published results across all 11 datasets and GROVER base on 8 of 11, for a 6.1% mean relative improvement — 2.2% on classification, 10.8% on regression. GROVER large reached ROC-AUC 0.940 on BBBP, 0.944 on ClinTox, 0.894 on BACE and 0.831 on Tox21, and RMSE 1.544 on FreeSolv, 0.831 on ESOL and 0.560 on Lipo. The largest gain, 23.9% relative, came on FreeSolv, which has only 642 labelled molecules. An ablation against the same architecture trained from scratch attributes a 3.8% mean AUC increase to pretraining.

#Applications

The pretrained encoder is most useful where labelled chemistry is scarce: ADMET and toxicity triage, solubility and lipophilicity estimation, and virtual screening where a few hundred measured compounds have to generalise to a much larger library. Teams either fine-tune the released checkpoint on their own assay data or run the fingerprint mode and feed the resulting vectors into a model their pipeline already uses, slotting the encoder in as one descriptor family alongside physicochemical or image-derived features.

#Impact

GROVER established graph-level self-supervised pretraining as a competitive route to molecular representation and remains a standard reference point in the area — it appears as a frozen descriptor source in SynGlue and as a comparison baseline in AmesNet and SMI-Editor. The practical caveats are those of an early, unmaintained release: the repository pins Python 3.6.8, the weights are distributed through OneDrive and Google Drive rather than a versioned model hub, and non-determinism in one indexing operation prevents exact reproduction of fine-tuning, which is why the authors also publish fine-tuned checkpoints for each of the 11 benchmark datasets. The encoder works purely on 2D molecular topology, so conformer- and geometry-dependent properties fall outside what it can represent.