CompGPT

Generative models for molecular design have mostly been trained on curated benchmark sets — MOSES and GuacaMol each hold fewer than two million molecules. That is a vanishingly small slice of accessible chemistry, and a model that has only ever seen it tends to generate variations on what it already knows rather than genuinely new scaffolds. The second habit that limits these models is atom-level tokenization: encoding a SMILES string one character at a time treats every atom as independent and gives the network no notion that a benzene ring or a sulfonamide is a single chemical idea.

CompGPT — the Compound Generative Pre-trained Transformer — from Tze Shin Chen, Jhih Wei Chu, and Jinn Moon Yang at National Yang Ming Chiao Tung University attacks both. It is a GPT-2 decoder pretrained on 200 million SMILES sampled from ZINC20, two orders of magnitude more chemistry than the standard benchmark corpora, and it reads those strings through a fragment vocabulary rather than an atom vocabulary. A Frequent Consecutive Subsequence pass over the training set identifies 3,151 recurring substructures — functional groups, ring systems, common linkers — and each becomes a single token. Sequences get shorter, and the units the model manipulates correspond to things a chemist would name.

Alongside generation, the authors treat interpretability as a first-class goal, visualizing attention across the model's 12 layers and 12 heads to show what chemical structure each depth is attending to. The work was posted to bioRxiv in January 2025 under a CC-BY 4.0 license, with code and a trained checkpoint on GitHub. It belongs to the same family of ZINC-scale chemical language models as FragAtlas-62M, which applies the approach to fragments rather than whole drug-like compounds.

#Key Features

• Substructure tokenization: A learned vocabulary of 3,151 frequent fragments replaces per-atom encoding, shortening sequences while preserving chemically meaningful units.
• ZINC-scale pretraining: Training on 200 million compounds gives coverage of commercially accessible chemistry far broader than benchmark-set training affords.
• Interpretable attention: Attention weights across layers show a progression from global positional relationships, through local features such as aromatic rings and functional groups, to how moieties connect into a scaffold.
• Adaptable base checkpoint: The generically pretrained model can be fine-tuned toward a chemical series, demonstrated on EGFR and FGFR3 inhibitor chemistry where generated compounds reproduced moieties matching known binding contacts including EGFR Met793.

#Technical Details

The architecture is a standard GPT-2 decoder: 12 transformer layers, 12 attention heads with 64-dimensional keys and values, a 768-dimensional embedding, and a 1,024-token context. Training used the Adam optimizer at a learning rate of 2e-5 for 10 epochs, with generation at temperature 1.0.

A corpus-size ablation trained the same architecture on 20M, 40M, 90M, 150M, and 400M compounds. Uniqueness held at 99.9% throughout, while validity and novelty climbed together — 78.7% and 78.7% at 20M, rising to 99.2% and 99.0% at 400M — with the marginal gain past 150M small enough that the authors settled on 200M as the coverage-versus-compute trade-off for the released model. An encoding ablation isolates the tokenizer's contribution: moiety-based encoding reached 99.6% validity, 100% uniqueness, 99.9% novelty, and internal diversity of 0.881 and 0.872, against 92.3%, 99.9%, 93.0%, 0.856, and 0.832 for atom-level encoding. Benchmarked on MOSES against CharRNN, VAE, AAE, MolGPT, LatentGAN, and JT-VAE over 10,000 generated SMILES, CompGPT reached 99.5% validity, 99.2% uniqueness, and 98.8% novelty, the highest novelty and internal diversity in the comparison; JT-VAE reached 100% validity by checking chemical correctness at every generation step.

#Applications

CompGPT is aimed at the earliest stage of small-molecule discovery: enumerating structurally diverse, chemically valid starting points across a much wider swathe of accessible chemistry than a benchmark-trained generator explores. Because the base checkpoint is generic, teams can fine-tune it toward a target class — as the authors did for kinase inhibitors — and use the attention visualizations to check that the model is attending to the substructures that matter for the series rather than producing plausible-looking strings for opaque reasons.

#Impact

The paper's central empirical claim is that both training-corpus scale and tokenization granularity carry measurable weight in molecular generation, and the two ablations separate those effects cleanly enough to be useful to anyone building a chemical language model. Evaluation is confined to distribution-level metrics — validity, uniqueness, novelty, internal diversity — with no downstream property prediction, binding assay, or synthesis; the kinase analysis is a structural-similarity comparison against known inhibitors rather than an activity result. The GitHub repository carries the training code, the fragment tokenizer, and a pretrained checkpoint, but no license file, so reuse terms are undefined. The preprint has not been peer reviewed.