SMI-Editor

SMI-Editor is a pretrained SMILES language model from Peking University with collaborators at the International Digital Economy Academy, Sichuan University and the University of Washington, published at ICLR 2025.

SMILES strings are the standard textual encoding of molecular structure, and the obvious way to pretrain on them is masked language modeling, as in ChemBERTa and related SMILES encoders. The paper's starting observation is that this transfers badly from natural language. A SMILES token is a single atom, bond, or bracket, whereas chemical function lives in substructures — a carboxyl group, a ring system — so predicting one masked character teaches very little chemistry. The authors show the consequence empirically: a SMILES masked LM exceeds 90 percent mask-prediction accuracy within 5,000 training steps and over 95 percent by 10,000, at every model size down to 6.7 million parameters, and raising the mask ratio from 15 to 45 percent does not delay it. The task saturates, so capacity goes unused. A second problem is a train-inference mismatch: these models only ever see corrupted strings containing [MASK] symbols, which never appear in real SMILES.

SMI-Editor replaces masking with editing. A molecule is fragmented, a random subset of fragments is discarded, the remainder is reassembled into a valid SMILES string, and the model is trained to edit that string back into the original. Because whole fragments go missing, the supervision signal is fragment-level; because the input is a real molecule rather than a masked one, the model always sees valid chemistry.

#Key Features

• Fragment-level supervision: Molecules are split with a modified RMCF variant of the BRICS algorithm, which further breaks rings and side chains into smaller fragments, and a random subset is dropped before reassembly. Recovering them requires reasoning about substructures rather than individual atoms.
• Valid SMILES as input: The corrupted molecule is a real, parseable molecule, so no artificial [MASK] token ever enters the input distribution and training matches inference.
• Three editing heads: A deletion head decides per token whether to keep or remove, a placeholder-insertion head predicts where and how many tokens to add as a 256-class problem per position, and a token-prediction head fills each placeholder.
• Dual deletion loss: Standard edit-based models supervise deletion only on tokens the model itself inserted; SMI-Editor adds supervision for removing erroneous tokens present in the corrupted input, which the ablations show is the second most important training signal after placeholder insertion.
• Demonstrable substructure sensitivity: Deleting hydrophilic groups and deleting random atoms shift SMI-Editor's predictions differently on ESOL and FreeSolv, whereas a masked LM responds to both perturbations alike.

#Technical Details

The encoder is a 12-layer Transformer with hidden size 768, feed-forward dimension 3072 and 12 attention heads, totaling 86.3 million trainable parameters, implemented as a Levenshtein-transformer architecture in fairseq. Pretraining uses the 19-million molecule ligand corpus assembled for Uni-Mol, a 114.76 GB dataset, with a fragment drop ratio of 0.15. Downstream evaluation is on seven MoleculeNet classification tasks under scaffold splitting, reported as ROC-AUC: BACE 80.3, BBBP 77.4, Tox21 77.1, SIDER 63.0, MUV 80.2, ClinTox 98.9 and ToxCast 67.4, for a mean of 77.8. That is the best average in the comparison, ahead of MolCLR at 75.5, GraphMVP at 73.3, Mole-BERT at 73.4 and GROVER at 71.5, and ahead of the 3D representation models 3D InfoMax (71.8) and MoleculeSDE (74.5). A control model, SMI-MLM, trained on identical data and architecture with a masked objective, reaches 73.2. Ablations replacing fragment dropping with random atom dropping or random token masking cost 1.9 and 2.7 mean points respectively.

#Applications

SMI-Editor produces molecular representations for property prediction — solubility, blood-brain barrier penetration, toxicity endpoints — from SMILES alone, with no conformer generation or 3D coordinates required. That makes it cheap to apply across large virtual libraries in early-stage drug discovery, where computing 3D structures for millions of compounds is the bottleneck. The repository ships under MIT with the pretraining command, the architecture, and a checkpoint download, and recommends the Uni-Mol downstream evaluation protocol for fine-tuning on new endpoints.

#Impact

The result of practical interest is that a 1D SMILES model with the right pretraining objective outperforms several models built on 3D molecular geometry, which suggests the spatial information those models add is partly recoverable from the string when the objective is hard enough to force it. The diagnostic half of the paper — showing that SMILES masked language modeling saturates within thousands of steps regardless of scale — is a useful negative result for anyone considering scaling a SMILES MLM. The main practical caveat is distribution: the released checkpoints are hosted on a Google Drive link rather than a model hub, and while the repository carries an MIT license there is no separate license statement attached to the weights themselves.