MoleSG

The same molecule is two different objects depending on how you write it down. As a SMILES string it is compact text — good for retrieval, and readable by any Transformer borrowed from NLP — but ring closures and branch points scatter chemically adjacent atoms far apart in the sequence. As a graph it puts bonds and neighborhoods where they belong, but the message-passing layers that read graphs gather information locally and struggle to see across a whole scaffold. Most self-supervised pretraining for small molecules commits to one view and inherits its blind spot.

MoleSG takes both, and makes each one responsible for what the other cannot see. During pretraining it masks a fraction of atoms in the graph, then masks SMILES tokens drawn only from the atoms left unmasked in the graph — a non-overlapping mask, built from an explicit atom-index correspondence between the two representations. Neither view retains the answer to its own reconstruction problem, so the only way to recover a hidden atom is to read the other modality. The two masked inputs are embedded by modality-specific encoders, concatenated, and passed through one shared Transformer backbone, with separate decoders reconstructing the original SMILES and graph.

The framework comes from the Digital Medical Research Center at Fudan University, published in Briefings in Bioinformatics in 2024. Its code borrows the SMILES tokenizer and vocabulary treatment from ChemBERTa and the graph encoder from CoMPT; the name MoleSG comes from an earlier, withdrawn ICLR 2024 submission of the same work, and only the journal version's title drops it.

#Key Features

• Non-overlapping masking: Masked regions in SMILES and graph are constructed to be disjoint, which turns reconstruction into a cross-modal task rather than two independent in-modality fill-in-the-blank problems.
• One backbone, two modalities: Rather than a two-tower encoder joined by a contrastive loss, both token streams enter a single Transformer, letting attention operate between individual SMILES tokens and individual atoms.
• Corpus-wide pretraining, one checkpoint: A single pretraining run over unlabeled molecules produces one set of weights that is then fine-tuned on every downstream task; the authors distribute that checkpoint.
• The graph encoder is what transfers: Fine-tuning reuses only the pretrained graph encoder with a task head. The shared backbone is trained on paired inputs and is not used downstream, where a single modality is supplied.

#Technical Details

Pretraining uses 250,000 unlabeled molecules sampled from ZINC15, run for 90,000 iterations with AdamW at a base learning rate of 1e-3. SMILES tokens are embedded by a RoBERTa-style stack of multi-head attention blocks over a vocabulary extended to cover the corpus; graphs go through CoMPT, whose communicative kernel passes messages between node and edge states. Mask ratios were tuned per modality — 25% of graph nodes and 15% of SMILES tokens.

Evaluation covers 14 MoleculeNet datasets comprising 678 binary classification tasks and 19 regression tasks, split 8:1:1 by scaffold (random for QM9) and averaged over three seeds. On classification, MoleSG reaches 97.9 ROC-AUC on BBBP, 99.1 on ClinTox, 95.1 on BACE, 85.0 on Tox21, 74.2 on ToxCast, 70.0 on SIDER, 85.1 on MUV and 87.7 on HIV. On regression it reports 0.599 RMSE on ESOL, 0.932 on FreeSolv, 0.545 on Lipophilicity and 29.6 MAE on QM7, ahead of the knowledge-augmented baseline KANO on every one. Ablations sweeping the mask overlap ratio, and a comparison against the same network pretrained on a single modality, isolate the contribution of the masking scheme: graph-only pretraining moves BBBP from 96.1 to 96.8, while the cross-modal objective reaches 97.9.

#Applications

The targets are the standard early-discovery filters — blood-brain-barrier penetration, toxicity panels, aqueous solubility, lipophilicity, binding to BACE — where labeled molecules number in the thousands and a pretrained representation is worth more than a larger supervised model. A team screening a compound series fine-tunes the released graph encoder on its own assay readouts, supplying molecules as graphs alone; the SMILES branch exists only to shape the representation during pretraining.

#Impact

MoleSG's contribution is the masking scheme more than the checkpoint: the demonstration that enforcing disjoint masks across two views of the same object, with an index correspondence to make "disjoint" well defined, extracts more than masking each view independently, and more than the contrastive alignment used by earlier SMILES-plus-graph methods. That idea generalizes past chemistry, and the authors propose extending it to proteins, which likewise have sequence and graph forms — a proposal, not a demonstrated capability of this model. Its limits are worth stating plainly: the 250,000-molecule pretraining corpus is small next to the 11 million compounds behind GROVER or the 1.1 billion behind MoLFormer-XL; evaluation never leaves the MoleculeNet property-prediction panel; and the repository carries no license file, with weights and processed data distributed as Google Drive archives rather than through a model hub.