ChemLML

Ask for "a steroid ester that inhibits acetylcholinesterase" and a text-to-molecule model should hand back a SMILES string. Most systems that do this are trained end to end on paired text–molecule corpora, which discards two bodies of pretrained knowledge at once: everything a scientific language model already knows about chemical prose, and everything a molecular language model already knows about valid, drug-like structures. Retraining both from scratch is expensive and caps how large either half can be.

ChemLML — Chemical Language Model Linker — keeps both halves and trains only the seam between them. A single cross-attention adapter sits at the last layer of a frozen pretrained molecule decoder. Text embeddings from a pretrained scientific text encoder are projected into the molecule model's embedding dimension, and the adapter mixes them into the molecule token embeddings before the language-modelling head. Because the refined embeddings still live in the molecule decoder's own space, generation stays inside the chemical vocabulary the decoder was pretrained on instead of drifting into natural-language tokens — an architecture analogous to LM-Design and ProtT3 in the protein domain.

The model comes from Yifan Deng, Spencer S. Ericksen and Anthony Gitter at the University of Wisconsin–Madison and the Morgridge Institute for Research, the group behind METL and Protein Set Transformer. It was posted to arXiv in October 2024 and published in Journal of Chemical Information and Modeling in August 2025.

#Key Features

• Adapter-only training: The molecule decoder is always frozen and the text encoder is optionally frozen, so a working configuration trains as few as 4.7M parameters out of 464M total — roughly a fiftieth of MolT5's 248M trainable parameters, which it still outperforms.
• Swappable backbones: The paper pairs SciBERT, the Text+Chem T5 encoder, and Galactica at 125M, 1.3B and 6.7B against MolGPT, MolGen, MolGen-7B and MolXPT decoders, making ChemLML a family of configurations rather than one checkpoint.
• SMILES beats SELFIES for conditional generation: In a matched MolGPT comparison, the SMILES model reaches 0.454 Morgan fingerprint Tanimoto similarity against 0.308 for SELFIES, and the gap survives restricting the comparison to molecules both representations render validly.
• A filtered PubChem generation test set: Many PubChem descriptions are generic or map to several molecules. Filtering to descriptions over 30 words with one-to-one molecule pairings, then removing the ChEBI-20 overlap, leaves 11,563 examples from which 3,000 form the released test set.
• Output stays chemical: Generating 100 unique membrane-permeable candidates, ChemLML with MolGen passed all validity filters 98.0% of the time, against 43.7% for MolT5 and 10.6% for Text+Chem T5, whose failures included natural-language tokens and salt fragments mixed into the SMILES.

#Technical Details

Training and evaluation use ChEBI-20, 33,010 molecule–description pairs split 80/10/10, with a Noam optimizer and 4,000 warm-up steps. On the ChEBI-20 test set the strongest configuration is a fine-tuned T5 encoder with MolXPT: 114M trainable of 464M total parameters, 0.389 exact match, 0.868 MACCS FTS and 0.727 Morgan FTS. With the text encoder frozen, T5 encoder + MolXPT still reaches 0.305 exact match and 0.674 Morgan FTS on 4.7M trainable parameters. On the PubChem-filtered set the same fine-tuned pairing leads at 0.286 exact match and 0.453 Morgan FTS. ChemLML also returns exactly matching molecules more often than the MolT5 and Text+Chem T5 baselines, which the authors attribute to the molecule decoders' large pretraining corpora.

#Applications

The obvious use is early-stage hit generation: describing a desired chemotype or target in prose and sampling candidates rather than screening a library. The paper works two case studies. For eight protein targets — AChE, IMPDH, HSP90AA1, the SARS-CoV-2 main protease, LSD1, TOPIIB, ACE and MAPKK1 — generated compounds were docked with four programs under a consensus score; on four targets the median ChemLML score exceeded that of the known ground-truth ligand. A second study prompted for drug-like compounds with high passive permeability and scored the output with an MDR1-MDCK efflux ratio predictor.

#Impact

ChemLML's contribution is a recipe rather than a single artifact: any pretrained text encoder and any pretrained molecular decoder can be joined for the cost of training one adapter, which makes text conditioning cheap to re-derive as either backbone improves. The filtered PubChem test set is a useful side product, since evaluating on unfiltered PubChem descriptions rewards models for guessing from ambiguous text. The honest limits are stated in the paper: ChemLML does molecule generation only, where MolT5 and Text+Chem T5 are multi-task; larger Galactica models proved hard to fine-tune; case-study assessment is docking and predicted permeability rather than assay data; and ChEBI-20 and PubChem descriptions follow a fixed format unlike the prompts a working chemist would write. Code is MIT-licensed and the trained adapters are downloadable, so the configurations can be reproduced directly.