AMPGPT & MDHGPT

Sequence-based generative models for protein design are almost always trained to reproduce their training sequences amino acid by amino acid. That objective has a known flaw: two single-residue substitutions from the same wild-type sequence incur identical loss, even when one preserves function and the other destroys it. The discrete amino acid space is also highly multimodal, which makes the underlying distribution hard to fit.

AMPGPT and MDHGPT are the two model families produced by a paper from the Machine Biology Group at the University of Pennsylvania that attacks this problem directly. Both are fine-tuned from ProGen2-large, the 2.7-billion-parameter autoregressive protein transformer, but they are trained against a second objective in addition to the usual cross-entropy: the distance between the generated sequence and its training template as measured in the latent space of a frozen ESM-2 encoder. AMPGPT generates antimicrobial peptides; MDHGPT generates malate dehydrogenases, a much longer and more structurally constrained enzyme class.

Because embeddings from a masked protein language model cluster functionally similar sequences together, that latent distance acts as an approximate function checker. The paper frames the scheme equivalently as knowledge distillation, with the encoder as teacher and the generator as student.

#Key Features

• Latent-space likelihood as a second objective: The total loss adds a mean squared error term between the encoder embeddings of the generated and template sequences, weighted by a hyperparameter that the authors sweep over 1, 10, and 100.
• Policy gradient bridges the non-differentiable gap: Because the latent error cannot be backpropagated through discrete sampling, the generator is treated as a policy and the encoder as the environment, which reduces in practice to reweighting each sample's ordinary gradient by one plus the scaled latent error.
• Bidirectional decoding: Training on reversed sequences as well as forward ones lets the models generate from the N-terminus, from the C-terminus, or from an even mixture, with each mode giving a different diversity and similarity profile.
• Reward baseline variants: EMA variants subtract an exponential moving average of the historical reward, a standard variance-reduction trick that yields the strongest enzyme results.
• Sequence novelty is a tuned target, not an accident: Generations are held to a 60–80% similarity band against training data, chosen to balance exploration against staying inside the functional distribution.

#Technical Details

The generator is initialized from ProGen2-large and the frozen validator is the 33-layer ESM-2 encoder. AMPGPT trained on 42,210 experimentally verified peptides pooled from APD3, DRAMP, LAMP2, DBAASP, and dbAMP for 50 epochs at batch size 16; MDHGPT trained on 16,706 enzyme sequences for 30 epochs at batch size 4. Both used AdamW at a 1e-5 peak learning rate with a warm-up over the first 5% of steps. Generation is seeded by sampling terminal dipeptides for peptides and decapeptides for enzymes from the training frequency distribution.

Evaluation used external predictors rather than the training objective. Over 10,000 generated peptides scored by the activity regressor APEX, AMPGPT variants reached a mean median MIC of 95.9–99.6 μmol/L against 102.7–104.9 for the same architecture trained without the latent objective, and 104.2–129.8 for GAN, VAE, conditional-VAE, and diffusion baselines. The paper's own tables show the gain is confined to APEX: on the Macrel classifier the ablated model is marginally ahead in several cells, and the authors report those differences as not statistically significant. The enzyme results are far less ambiguous — MDHGPT reaches 96.8% of generations classified as malate dehydrogenases versus 0% for the ablated model and for four ProtGPT checkpoints fine-tuned on sequence clusters.

#Applications

The immediate use is candidate generation for antibiotic discovery, where a pool of tens of thousands of novel peptides can be filtered by an activity regressor down to a synthesis-scale shortlist, and for enzyme engineering, where the malate dehydrogenase result shows the approach carries from 1–100 residue peptides to full-length catalytic proteins. More broadly, the training scheme is architecture-agnostic: any generator that explicitly models training-data likelihood — variational autoencoders, flow models, diffusion models — can adopt the same reweighting term as a drop-in addition to its loss.

#Impact

The paper's contribution is a training recipe more than a product, and its clearest evidence is the enzyme ablation, where adding the latent objective moves the fraction of functional generations from zero to nearly all. Validation is entirely computational — no generated peptide or enzyme was synthesized or assayed, and the authors name wet-lab confirmation as future work — so the reported gains rest on the reliability of Macrel, APEX, and a purpose-built enzyme classifier. Neither checkpoint has been released, and no public code accompanies the preprint, which remains unreviewed and licensed CC BY-NC. What the work establishes is narrower but useful: that a frozen protein language model can supply a differentiable-enough functional signal during generative training, without any labelled activity data.