Peptide language model that generates antimicrobial, anticancer, and target-binding sequences, adapted per task by Mixture-of-Experts plugins.
No providers recorded yet. Browse all providers
PepGenWL is an autoregressive language model for generating bioactive peptides. Peptide therapeutics sit in an attractive middle ground between small molecules and antibodies, and generative sequence models are an obvious tool for exploring their design space — but peptides are conformationally flexible in a way that folded proteins are not. A standard autoregressive decoder predicts residue i+1 strictly from residues 1 through i, baking a rigid left-to-right ordering into the model. For a short, floppy chain whose residue-to-residue contacts are not well described by sequence adjacency, that prior is stronger than the biology warrants.
The model's answer is weak order dependence: tolerance for out-of-order input, introduced as an explicit inductive bias. An "induction block" — a transformer decoder variant that takes the position of the next predicted residue as an independent position encoding rather than inferring it from the running sequence — sits atop the stack, so the model can be prompted with permuted residues and still generate coherently. The work comes from Peking University's Shenzhen campus with Pengcheng Laboratory, Shenzhen Bay Laboratory, and the University of Shanghai for Science and Technology. It appeared as a preprint in January 2024 under the name PepGenWOA and was published in the Journal of Chemical Information and Modeling in 2025 as PepGenWL.
Where peptide language models such as PepBERT supply representations for downstream classifiers, PepGenWL is generative end to end: one pretrained base covers antimicrobial peptides, anticancer peptides, and target-conditioned binders.
The generator is a stack of 24 transformer decoder blocks including the induction block, with a hidden size of 1,024, a 4,096-unit MLP intermediate size, and 16 attention heads — roughly 350 million parameters. Pretraining used about 140 million natural protein sequences under 128 residues drawn from BFD, Pfam and UniProt, so that the base model captures the grammar of short natural sequences; training ran on 128 Ascend-910 NPUs for about ten hours with Adam, a 1e-4 learning rate, cosine decay, and cross-entropy loss.
For antimicrobial and anticancer generation the model was fine-tuned on 20,129 AMP and 4,381 ACP sequences compiled from public databases, and scored on alignment score (novelty relative to the training set), instability score, and a classifier-predicted bioactivity probability. Against the strongest baseline, MMCD, PepGenWL improves AMP alignment score by 12.1%, instability by 12.6%, and antimicrobial probability by 5.8%; for ACPs the margins are 14.7%, 11.7%, and 8.3%. The binder case study targets the SARS-CoV-2 Omicron BA.5 receptor-binding domain, fine-tuning first on a stable miniprotein scaffold set and then on more than 160,000 Rosetta-designed candidate binders. Five adaptation strategies were compared — full-parameter, frozen-layer, adapter tuning, LoRA, and LoRAMoE — with LoRAMoE giving the best balance of retained general knowledge and new-task performance.
The model suits early-stage therapeutic peptide discovery, where the bottleneck is choosing which of an enormous designable space to synthesize. The antimicrobial and anticancer modes produce candidate sequences with high predicted bioactivity and low similarity to known peptides; the binder mode takes a structural target and yields sequences already shaped by stability and binding constraints, feeding directly into structure-based screening. The SMILES extension broadens this to chemically modified and cyclic formats that matter for peptide drug stability and oral availability.
PepGenWL treats peptide generation as a lifelong-learning problem rather than a series of independent fine-tunes, and the LoRAMoE comparison is the substantive contribution: it measures the stability-plasticity tradeoff directly rather than assuming full fine-tuning is adequate. The wet-lab validation is real but narrow — one target, seven synthesized candidates — so the 28.6% binding rate is best read as a proof of pipeline rather than a general hit rate. The released code is Apache-2.0 but written for MindSpore on Ascend-910 hardware, and ships training and generation scripts without pretrained checkpoints, so reproducing the model means re-running pretraining on the cited public databases.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.