Chemical language model pretrained on up to 77 million PubChem SMILES strings for molecular property prediction on the MoleculeNet benchmarks.
ChemBERTa asks whether the pretraining recipe that reshaped natural language processing carries over to molecules written as text. Labeled data for molecular property prediction is expensive — every measurement is a laboratory experiment — while unlabeled SMILES strings are available by the hundreds of millions. Released in October 2020 by researchers at DeepChem, the University of Toronto, and Reverie Labs, it was one of the first systematic evaluations of BERT-style masked language modeling against the graph neural networks and hand-crafted fingerprints that dominated cheminformatics.
The model treats a SMILES string as a sentence, pretrains a RoBERTa encoder to recover masked tokens, and then fine-tunes the learned representation on downstream tasks from the MoleculeNet suite. The original paper's central result was a scaling relationship rather than a benchmark win: downstream accuracy climbed steadily with pretraining set size, gaining a mean 0.110 ROC-AUC when the corpus grew from 100K to 10M molecules, even though the model still trailed the Chemprop D-MPNN baseline on most tasks. The authors also curated and released 77 million canonicalized PubChem SMILES for community pretraining.
ChemBERTa-2, published in 2022 by a team spanning Reverie Labs, MIT CSAIL, UC Berkeley, and Deep Forest Sciences, revisited the pipeline with the full 77M corpus, a hyperparameter search, and a second pretraining objective based on computed molecular descriptors. Those changes closed most of the gap to graph models, and the family is now a standard SMILES-encoder baseline alongside MoLFormer-XL and later encoders such as MolDeBERTa.
Both generations are RoBERTa encoders built on HuggingFace Transformers. The original uses 6 layers and 12 attention heads with a 52,000-token byte-pair vocabulary and a 512-token maximum sequence, masking 15% of tokens; pretraining on the 10M-compound PubChem subset took roughly 48 hours on a single V100 GPU. ChemBERTa-2 swaps the byte-pair vocabulary for a 591-token dictionary of SMILES characters, samples 50 random architecture configurations between 5M and 46M parameters, and trains the five best on 5M, 10M, and 77M-compound subsets under both masked language modeling and multi-task regression. Scaling the corpus from 5M to 77M unique SMILES improved pretraining loss by 25–35%, and pretraining loss transferred to downstream error close to linearly on lipophilicity, less predictably on BACE classification.
On MoleculeNet, ChemBERTa-2 beat the Chemprop D-MPNN on six of eight tasks. The 77M multi-task-regression model reached 0.798 RMSE on Lipophilicity against 1.212 for D-MPNN, and 0.889 on Delaney solubility against 1.105; the 5M multi-task model reached 0.834 ROC-AUC on the Tox21 SR-p53 task against 0.719. ClinTox is the clear weak spot, at 0.601 ROC-AUC versus 0.906 for the graph baseline.
Medicinal chemists and computational chemistry teams use ChemBERTa as a molecular featurizer: embed a compound library once, then fit a small head for aqueous solubility, lipophilicity, blood-brain-barrier permeability, clinical toxicity, or on-target inhibition using the few hundred to few thousand labeled examples a typical assay produces. Because the checkpoints are small, permissively licensed, and reachable through both HuggingFace and DeepChem's featurizer API, it is a common first baseline in ADMET modeling, a teaching example for transfer learning in cheminformatics, and a backbone offered by hosted fine-tuning services for SMILES-to-label workflows.
ChemBERTa established SMILES masked language modeling as a credible alternative to graph neural networks for molecular property prediction, and its scaling analysis anticipated the billion-molecule chemical language models that followed. Its checkpoints appear routinely as baselines in later SMILES-encoder work, and the curated PubChem-77M corpus became a shared pretraining resource. The limitations are real: both papers are preprints rather than peer-reviewed articles, with ChemBERTa-2 presented at the 2021 ELLIS Machine Learning for Molecule Discovery workshop; the models encode no 3D geometry and expect canonicalized SMILES; accuracy on small, imbalanced classification sets such as ClinTox lags graph baselines by a wide margin; and the ChemBERTa-2 HuggingFace repositories ship without model cards, so architecture and training details must be read from the paper.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.