AMPGPT

Sequence-based generative models for protein design are almost always trained to reproduce their training sequences amino acid by amino acid. That objective has a known flaw: two single-residue substitutions from the same wild-type sequence incur identical loss, even when one preserves function and the other destroys it.

AMPGPT, from the Machine Biology Group at the University of Pennsylvania, generates antimicrobial peptides under a training scheme built to close that gap. It is fine-tuned from ProGen2-large, the 2.7-billion-parameter autoregressive protein transformer, but trained against a second objective alongside the usual cross-entropy: the distance between the generated sequence and its training template as measured in the latent space of a frozen ESM-2 encoder.

Because embeddings from a masked protein language model cluster functionally similar sequences together, that latent distance acts as an approximate function checker — a signal about whether a generation is still the kind of thing it was meant to be, obtained without any labeled activity data. The paper frames the scheme equivalently as knowledge distillation, with the encoder as teacher and the generator as student. The same recipe produced MDHGPT, which applies it to a full-length enzyme class instead of short peptides.

#Key Features

• Latent-space likelihood as a second objective: The total loss adds a mean squared error term between the encoder embeddings of the generated and template sequences, weighted by a hyperparameter the authors sweep over 1, 10, and 100.
• Policy gradient bridges the non-differentiable gap: Because the latent error cannot be backpropagated through discrete sampling, the generator is treated as a policy and the encoder as the environment, which reduces in practice to reweighting each sample's ordinary gradient by one plus the scaled latent error.
• Bidirectional decoding: Training on reversed sequences as well as forward ones allows generation from the N-terminus, from the C-terminus, or from an even mixture, each mode giving a different diversity and similarity profile.
• Novelty as a tuned target: Generations are held to a 60–80% similarity band against training data, chosen to balance exploration against staying inside the functional distribution.

#Technical Details

The generator is initialized from ProGen2-large and the frozen validator is the 33-layer ESM-2 encoder. Training used 42,210 experimentally verified peptides pooled from APD3, DRAMP, LAMP2, DBAASP, and dbAMP, for 50 epochs at batch size 16, with AdamW at a 1e-5 peak learning rate and warm-up over the first 5% of steps. Generation is seeded by sampling terminal dipeptides from the training frequency distribution.

Evaluation used external predictors rather than the training objective. Over 10,000 generated peptides scored by the activity regressor APEX, variants reached a mean median MIC of 95.9–99.6 μmol/L, against 102.7–104.9 for the same architecture trained without the latent objective and 104.2–129.8 for GAN, VAE, conditional-VAE, and diffusion baselines. The paper's own tables show that gain is confined to APEX: on the Macrel classifier the ablated model is marginally ahead in several cells, and the authors report those differences as not statistically significant.

#Applications

The immediate use is candidate generation for antibiotic discovery, where a pool of tens of thousands of novel peptides can be filtered by an activity regressor down to a synthesis-scale shortlist. Bidirectional decoding gives some control over that pool's character, trading diversity against similarity to known actives depending on which terminus seeds generation. More broadly, the training scheme is architecture-agnostic: any generator that explicitly models training-data likelihood — variational autoencoders, flow models, diffusion models — can adopt the same reweighting term as a drop-in addition to its loss.

#Impact

The contribution is a training recipe more than a product, and on peptides specifically the evidence for it is mixed by the authors' own account: the improvement shows up under one external scorer and not under another, and they say so. The stronger case for the method comes from the enzyme sibling, where the ablation is unambiguous. Validation is entirely computational — no generated peptide was synthesized or assayed, and the authors name wet-lab confirmation as future work — so the reported gains rest on the reliability of APEX and Macrel. No checkpoint has been released, and no public code accompanies the preprint, which remains unreviewed and licensed CC BY-NC.