Loading…
Peptide representation model that DoRA-tunes ChemBERTa on 100,000 modified and bioactive peptide SMILES for therapeutic property prediction.
No providers recorded yet. Browse all providers
A head-to-tail cyclized peptide carrying an N-methylated backbone and a non-canonical residue has no representation in the twenty-letter alphabet that protein language models such as ESM-2 tokenize — and the modification is precisely the part that makes the molecule a drug. Chemical language models have no such blind spot, because a SMILES string encodes the cyclization and the unnatural side chain as ordinary atoms and bonds. But ChemBERTa and its relatives were pretrained on PubChem-scale libraries of drug-like small molecules, in which a forty-residue macrocycle is a far-tail outlier.
PepDoRA is a deliberately small answer to that gap. Rather than pretraining a peptide chemical language model from scratch, Leyao Wang, Rishab Pulugurta, Pranay Vure, Yinuo Zhang, Aastha Pal and Pranam Chatterjee at Duke University take the released ChemBERTa-77M-MLM checkpoint and shift it into peptide space with weight-decomposed low-rank adaptation (DoRA). DoRA factors each pretrained weight matrix into a magnitude vector and a direction matrix and tunes the two separately, learning a low-rank update on the direction while the magnitude adapts freely. The training signal is plain masked language modeling at a 15% masking rate over 100,000 peptide SMILES strings.
The design choice is easiest to read against PeptideCLM, which attacks the same problem by pretraining from scratch on modified-peptide SMILES alone. That buys peptide-specific tokenization at the cost of the broad physicochemical prior a general chemical model already carries, and of coverage for bioactive linear peptides built only from wild-type residues. PepDoRA bets the other way: keep the small-molecule prior, move it.
The backbone is compact: a three-layer RoBERTa encoder with a hidden width of 384 and 12 attention heads, on the order of three million parameters. The "77M" in its name counts the PubChem molecules it was pretrained on, not its weights. Adaptation ran on two NVIDIA A6000 GPUs with AdamW at a learning rate of 2 × 10⁻⁵, weight decay 0.01, batch size 2, and at most 10 epochs over an 80/20 split of the peptide corpus.
On PAMPA membrane permeability, evaluated under PeptideCLM's clustered-holdout protocol with five-fold cross-validation repeated five times, PepDoRA records the lowest mean RMSE among the three ChemBERTa fine-tuning strategies tested (DoRA, LoRA, final-layer unfreezing), PeptideCLM-23M, and the untuned ChemBERTa-77M-MLM. Against PeptideBERT on non-fouling and hemolysis classification the result is closer to a tie — accuracy 0.87 versus 0.87 and 0.80 versus 0.81, with PepDoRA slightly ahead on recall and F1 (0.70 versus 0.69; 0.37 versus 0.35). The clearest margin is in peptide–protein binding: a contrastive model over roughly 12,000 Propedia v2.3 and PepNN pairs, with ESM-2-650M supplying target embeddings, reaches 95.9% binary accuracy, 62.1% top-1 and 86.1% top-10% accuracy, against 47.3% top-1 for PeptideCLM-23M and 27.6% for the untuned backbone.
The intended use is featurization for peptide-therapeutic triage: ranking macrocycle libraries by predicted passive permeability, screening candidates for hemolytic risk and non-fouling behavior, and shortlisting binders against a named protein target without requiring a structure. Because a single encoder handles both modified and natural chemistries, one embedding pipeline serves campaigns that mix stapled peptides, cyclic scaffolds and linear leads. Third-party reuse follows the same pattern — the HydrAffinity binding-affinity model consumes PepDoRA as one of its frozen ligand encoders.
PepDoRA is a compact demonstration that parameter-efficient fine-tuning can move a small-molecule chemical language model into peptide space cheaply enough to be worth doing on two workstation GPUs, and it extends the Chatterjee lab's peptide-design line — moPPIt among them — with a representation layer rather than a generator. The caveats are real: it remains an unreviewed preprint at v1, the classification gains over PeptideBERT sit within run-to-run noise, no training or evaluation code accompanies the weights, and the CC BY-NC-ND 4.0 license on the adapter bars both commercial use and derivative works — an awkward fit for an artifact whose value is being built upon.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.