G2PT

Most modern molecular graph generators work through adjacency matrices, usually with discrete diffusion: the model starts from noise over every possible node pair and denoises toward a valid structure. That formulation buys permutation invariance, but it costs something too. The network must be permutation-invariant as well, which restricts it to graph neural network architectures, and each denoising step samples matrix entries independently, which makes the true joint distribution hard to capture without many steps.

G2PT — the Graph Generative Pre-trained Transformer — abandons the adjacency matrix. It represents a graph as a sequence of tokens: first the node set, then the edge set, with each edge written as a source, destination, and type. A standard decoder-only transformer then learns the distribution by next-token prediction, exactly as a language model does over text. Because a molecule is sparse, this encoding is far more compact than the matrix it replaces; on planar graphs the token sequence runs 737 tokens where the adjacency encoding needs 2,018.

The model was built by researchers at Tufts University, Northeastern University, and Cornell University, and presented at ICML 2025. Its formulation is domain-general — the paper also evaluates on generic and social-network graphs — but the molecular case is where it is developed furthest, with pretrained checkpoints on the QM9, MOSES, and GuacaMol corpora and fine-tuning recipes for both goal-directed design and property prediction. Unlike string-based chemical language models such as ChemBERTa, which operate on SMILES, G2PT tokenises the molecular graph itself, so connectivity is explicit rather than something the model must recover from a linearisation.

#Key Features

• Node-set / edge-set tokenisation: A sparse graph is encoded directly as a short token sequence rather than as a dense matrix, which shortens context, speeds training, and removes the architectural constraint to permutation-invariant networks.
• Next-token pretraining: Generation is ordinary autoregressive decoding, so the model inherits the tooling, scaling behaviour, and sampling controls developed for language models.
• Two fine-tuning routes to goal-directed design: Rejection sampling fine-tuning with self-bootstrapping progressively shifts the sampling distribution toward a property threshold, while PPO reinforcement learning optimises a token-level reward directly.
• Shared backbone for prediction: Swapping the token-prediction layer for an MLP head on the final token's hidden state converts the same pretrained model into a molecular property classifier.
• Released checkpoint family: Small, base, and large variants for each training corpus load through the standard causal-language-model interface and generate SMILES without additional training.

#Technical Details

Three sizes are trained: a small model with 6 layers, 6 attention heads, and d_model 384 (≈10M parameters); a base model with 12 layers, 12 heads, and d_model 768 (≈85M); and a large model with 24 layers, 16 heads, and d_model 1024 (≈300M). Molecular pretraining uses QM9, MOSES, and GuacaMol; four generic graph datasets cover the non-molecular case. Graphs are serialised under either breadth-first or degree-based node orderings, and released checkpoints exist for both.

On MOSES the model leads the compared state of the art on validity, uniqueness, Fréchet ChemNet distance, and nearest-neighbour similarity, and it performs strongly on GuacaMol and QM9. For goal-directed generation, the authors fine-tune the GuacaMol-pretrained model toward quantitative estimate of druglikeness, synthetic accessibility, and activity against glycogen synthase kinase 3 beta, using oracle functions from the Therapeutics Data Commons; rejection sampling with up to three rounds of self-bootstrapping shifts the distribution more decisively than PPO, which is held back by regularisation toward the base policy. On the eight MoleculeNet classification tasks — BBBP, Tox21, ToxCast, SIDER, ClinTox, MUV, HIV, and BACE — the base model averages 73.3 ROC-AUC against 64.9 for the identical architecture without pretraining, matching the strongest graph self-supervised baseline in the comparison. The model is trained purely on 2D graph structure, so it carries no 3D geometry or chirality information.

#Applications

The practical target is hit generation and lead optimisation, where a chemist has a scoring function — a docking score, a predicted ADMET property, a QSAR model — and wants synthesisable candidates biased toward it. The rejection-sampling recipe is attractive here because it needs only a scalar oracle and an acceptance threshold, with no gradient through the reward. The property-prediction path suits teams that want one backbone serving both generation and screening rather than two separate models.

#Impact

G2PT's contribution is largely representational: a compact sequence encoding lets an unmodified autoregressive transformer match or beat purpose-built graph diffusion models on molecular generation, bringing the accumulated engineering of language modelling to bear on a problem that had drifted toward specialised architectures. The absence of 3D and stereochemical information limits it for structure-based design, and the molecular benchmarks it reports are distributional rather than experimental. The training and sampling code carries the MIT license. The checkpoints do not: all thirteen download freely and load without gating, but each ships the auto-generated model card with no license declared, leaving reuse terms for the weights themselves undefined.