MLM-FG

Aspirin is written O=C(C)Oc1ccccc1C(=O)O in SMILES, and nearly everything a chemist cares about in that string lives in two short substrings: the ester near the front and the carboxylic acid at the end. A masked language model trained the conventional way — hide a random span of characters, predict it back — spends most of its budget reconstructing ring-closure digits and branch parentheses, and only occasionally has to reason about the groups that set solubility, toxicity, or target binding. MLM-FG changes what gets hidden rather than what gets read.

Built at Baidu Research with academic collaborators in China, the UK, Poland, and the United States, MLM-FG parses each training SMILES with RDKit's rule-based SMARTS patterns, locates the token clusters that correspond to functional groups, and masks those clusters instead of arbitrary spans. Recovering a masked carboxyl or amide from its surroundings is a chemistry problem, not a string-completion problem, so the pretraining signal pushes the encoder toward substructure-level understanding while the input stays ordinary, unmodified SMILES.

That last point separates the approach from the two families it competes with. Graph neural network methods inject explicit topology or 3D geometry, but conformers at pretraining scale have to be generated by force fields and are unreliable for flexible molecules. Fragment-tokenization methods change the input vocabulary itself, tying the model to a fragment library. MLM-FG keeps standard SMILES and standard architectures — it reuses the MoLFormer backbone and a RoBERTa encoder unchanged — and puts its chemical prior entirely into the masking rule.

#Key Features

• Functional-group-aware masking: RDKit SMARTS patterns identify functional groups in each SMILES string, and the masking targets those token clusters. The rule adapts to molecular complexity: atom masking when no groups are found, one group for molecules with fewer than ten, and a random 10% beyond that.
• Unmodified SMILES input: The tokenizer and sequence format stay standard, so the approach transfers to any SMILES encoder rather than requiring a fragment vocabulary.
• Two backbones, three data scales: Six pretrained checkpoints cover the MoLFormer and RoBERTa architectures at 10M, 20M, and 100M pretraining molecules, making the contribution of architecture and corpus size separable.
• Implicit geometry from sequence: Attention matrices between atomic tokens correlate with true interatomic 3D distances more closely than those of the same architectures pretrained with random masking — structural awareness acquired without ever seeing coordinates.

#Technical Details

Both variants use 12 transformer layers with 12 attention heads and a hidden dimension of 768, capping SMILES at 512 tokens; the MoLFormer variant has roughly 48.1M trainable parameters and the RoBERTa variant roughly 93.8M. Pretraining used 10, 20, or 100 million molecules randomly sampled from PubChem, on 16 NVIDIA V100 GPUs with AdamW, a batch size of 1024, and a learning rate of 3 × 10⁻⁵. Evaluation covers twelve MoleculeNet datasets under scaffold splits — seven classification tasks (BBBP, BACE, ClinTox, Tox21, SIDER, HIV, MUV) and five regression tasks (ESOL, FreeSolv, Lipophilicity, QM7, QM8) — where MLM-FG leads the reported baselines on nine of eleven benchmark comparisons. It reaches 0.9606 AUC-ROC on ClinTox against MoLFormer's 0.9451 and 0.7990 on MUV against 0.7599, and improves on the next-best ESOL result by 41%, despite a pretraining corpus roughly ten times smaller than MoLFormer's 1.1 billion molecules. The 3D-aware GEM remains ahead on BACE, SIDER, and QM8.

#Applications

The practical use is ranking compounds before anyone synthesizes them. Fine-tuned heads predict ADMET-flavored endpoints — blood-brain barrier penetrance, clinical toxicity, aqueous solubility, lipophilicity — from SMILES alone, which is what virtual screening of a purchasable library or a lead-optimization series needs. The frozen encoder is also usable directly: representations extracted for 312,879 downstream molecules without fine-tuning already separate by molecular weight, so the checkpoints serve as a molecular featurizer for clustering, similarity search, or as input to a smaller model where labels are scarce.

#Impact

MLM-FG makes a narrow, testable argument: much of the benefit attributed to explicit graph or 3D inputs can be recovered by choosing what to mask in a plain sequence model, at a fraction of the data-preparation cost. The ablations carry that argument — the same architectures pretrained with random subsequence masking or from scratch land well behind — and the attention analysis supplies a mechanism rather than just a score. The release is real but modest: the code is on GitHub and the six checkpoints are distributed through a Google Drive folder, with no license attached to either and no model hub presence. The authors are explicit that the model is predictive only, with no capability for reaction prediction or molecular generation, and that SMILES beyond 512 tokens are truncated. Scaling is also not monotonic — on ESOL the MoLFormer variant degrades as the corpus grows from 10M to 100M molecules — so the largest checkpoint is not automatically the right one for a task.