Chemformer

A computational chemistry group that wants to predict reaction products, run retrosynthesis, optimise a lead's properties and fit a QSAR model has historically needed four separate transformers — each trained from scratch on its own labelled dataset, each costing days per hyperparameter setting. The duplicated effort is in the fundamentals: every one of those models has to rediscover, from its own narrow task data, what a valid SMILES string looks like and which atoms bond to which.

Chemformer factors that work out. A BART encoder–decoder is pretrained on unlabelled SMILES with a denoising objective — corrupt a molecule's string, ask the decoder to reconstruct the original — and those weights are then loaded separately as the starting point for each downstream task. Because BART keeps its encoder and decoder stacks distinct, one checkpoint covers both shapes of problem: sequence-to-sequence tasks feed reactants to the encoder and generate products from the decoder, while discriminative tasks discard the decoder and put a small MLP head on the encoder's output. Encoder-only chemical language models cannot do the former, and models that concatenate input and output into one long sequence pay a quadratic memory cost for it.

The model comes from the Molecular AI group at AstraZeneca, with a co-author at Linköping University, and was published in Machine Learning: Science and Technology in 2022. The code was first released as MolBART — a name it still shares with MegaMolBART, NVIDIA's derivative of the same architecture.

#Key Features

• One pretraining run, many downstream tasks: the same weights initialise forward synthesis, retrosynthesis, molecular optimisation, property regression and bioactivity prediction, rather than each task funding its own pretraining.
• Chemistry-specific denoising objectives: alongside BART-style span masking, the model learns to map one SMILES form of a molecule to another by enumeration — a corruption task with no natural-language analogue — and on the two combined.
• Tunable on-the-fly augmentation: fine-tuning re-spells each input–output pair with a tunable probability, so only canonical data is stored and the model sees a different string form of the same molecule every epoch.
• Multi-task discriminative heads: task tokens prefix the input, letting one encoder cover three MoleculeNet property tasks at once, or bioactivity regression for 133 genes, instead of one model per endpoint.
• Served as a retrosynthesis expansion policy: a FastAPI service wraps a fixed checkpoint and answers single-step queries, which is how AiZynthFinder calls it during a route search.

#Technical Details

Chemformer is a pre-norm transformer with GELU activations in the BART layout. The base model has a dimension of 512, six layers and eight attention heads for roughly 45M parameters; Chemformer-Large widens this to 1024, eight layers and 16 heads for 230M. Its 523-token vocabulary comes from regex matching over ChEMBL 27 canonical SMILES, padded with spare slots that later become task tokens. Pretraining used 100 million molecules sampled from ZINC-15's roughly 1.5 billion — a deliberate subsample under compute constraints — for 1,000,000 steps on four V100 GPUs, taking about 2.5 days for the base model and six for the large.

On top-1 accuracy the pretrained models lead: 92.5% (base) and 92.8% (large) on USPTO-MIT separated forward prediction against 90.4% for the Molecular Transformer, and 53.6%/54.3% top-1 on USPTO-50K retrosynthesis against 48.3% for the Augmented Transformer. Twenty epochs of fine-tuning — under 30 minutes on one GPU — beat the previous SMILES-based best on that retrosynthesis benchmark. Elsewhere the picture is mixed: aggressive SMILES augmentation fills the beam with re-spellings of one molecule, so top-5 and top-10 accuracies trail graph-based methods, and on the discriminative side D-MPNN and MolBERT beat Chemformer on all three MoleculeNet tasks while a support vector regressor on Morgan fingerprints edges it out across the 133 ExCAPE bioactivity tasks.

#Applications

The strongest fit is reaction informatics: forward prediction for feasibility triage, and single-step retrosynthesis served as an expansion policy behind multi-step route planners. A separate fine-tune performs matched-molecular-pair optimisation, rewriting a molecule under tokens that request higher solubility, lower clearance or unchanged LogD. Groups with modest GPU budgets benefit most: starting from the released checkpoint replaces days of per-task training with hours.

#Impact

Chemformer established the pretrain-once, transfer-many pattern for SMILES models in an industrial drug-discovery setting, and its weights seeded later work: a disconnection-aware fine-tune that constrains which bonds a route may break, and evaluations of the seq2seq model as a multi-step retrosynthesis engine. Practical caveats matter for anyone picking it up: the checkpoints ship from a corporate Box share rather than a versioned, citable archive, and the original repository is archived in favour of AiZynthModels, whose loader needs a manual hyperparameter rename before it accepts the old files. The molecular optimisation results depend on in-house AstraZeneca property predictors that were never released, so that benchmark cannot be reproduced end to end outside the company.