Loading…
Chemical language model reading modified and cyclic peptides as SMILES, fine-tuned to predict passive membrane diffusion of macrocycles.
No providers recorded yet. Browse all providers
A macrocyclic drug candidate closed head-to-tail, with a fraction of its backbone amines N-methylated and a short polyethylene glycol chain at a free amine, is still a peptide — but none of that can be written in the twenty-letter alphabet a protein language model reads. Strip the modifications to fit the alphabet and what remains is the molecule that would not have crossed a membrane, because those modifications are what make it permeable. PeptideCLM abandons the residue alphabet instead: it reads peptides as SMILES strings, the same atom-level notation used for small molecules, so a ring closure, a D-amino acid, a non-canonical side chain, and a conjugated payload are all just characters in the input.
Developed by Aaron Feller and Claus Wilke at the University of Texas at Austin, it is a 44M-parameter BERT-style encoder pretrained by masked language modeling on 23 million molecules, then fine-tuned to predict passive membrane diffusion of cyclic peptides.
That places it between two established families. Small-molecule chemical language models such as ChemBERTa tokenize SMILES but were pretrained on drug-like compounds and capped context at 512 tokens — too short for a peptide written atom by atom. Protein language models handle length but not chemistry. PeptideCLM widens the window to 768 tokens, enough for roughly 100 residues. Its successor, PeptideCLM-2, scales the same idea to a much larger corpus and a family of checkpoints.
The architecture is a RoFormer-style BERT encoder with rotary position embeddings: 6 transformer blocks, 12 attention heads, 768 hidden dimensions, a 3072-wide feed-forward layer, and a 768-token context, totaling 44M parameters. Pretraining masks 15% of tokens under the standard 80/10/10 corruption scheme, with a batch size of 64 and a learning rate of 5 × 10⁻⁵, run across 8 AMD Radeon RX 5700 XT GPUs. The corpus combines 10M PubChem and 2.2M SureChEMBL small molecules with 825,632 SmProt peptides and 10M generated modified peptides. Downstream, a feed-forward regression head replaces the language modeling head and the whole network is fine-tuned on PAMPA log Pexp values from CycPeptMPDB. Averaged over six held-out clusters, the full model reaches 0.781 ROC-AUC and 0.738 PR-AUC, against 0.773 for the peptide-only ablation, 0.770 for the small-molecule-only ablation, 0.666 for a randomly initialized network of the same shape, and 0.743 for the strongest ChemBERTa-2 variant.
The intended use is peptide drug discovery, where a medicinal chemistry team needs a permeability or developability estimate for macrocycles that conventional protein models cannot represent. The checkpoints fine-tune on a few thousand labeled molecules for any per-molecule property, and the embeddings serve directly as features — the paper's own clustering analysis uses them that way, and Peptiverse adopts PeptideCLM-23M as its SMILES-side encoder alongside a protein language model.
PeptideCLM established that a modest chemical language model, pretrained on largely synthetic peptide chemistry, transfers usefully to a real pharmacological endpoint, and it has since become a standard SMILES-side baseline and building block in peptide property prediction. Its limitations are instructive: α-carbon chirality is unlearnable because stereochemistry was randomized during synthetic generation, so masked-prediction accuracy plateaus below what the small-molecule ablation reaches; 10 million of the 10.8 million pretraining peptides are synthetic rather than measured; the custom tokenizer cannot be loaded through the transformers library and must be pulled from the GitHub repository; and the HuggingFace model card is two sentences, leaving the paper as the substantive documentation. The Zenodo release also notes a cyclization bug in the SMILES used to train the published model, fixed in a later dataset version.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.