SaAMPLIFY

Protein language models score a mutation by how plausible the substituted residue is given the evolutionary record, not by whether the resulting protein would still fold. Closing that gap normally means a new architecture that consumes structures at inference time.

SaAMPLIFY is the AMPLIFY protein language model after a post-training recipe that closes part of the gap instead, transferring structural knowledge out of pretrained protein graph neural networks and into the existing sequence model. The same recipe applied to ESM-2 yields SaESM2, and running it on two unrelated backbones is how the paper establishes that the effect is a property of the method rather than of one model family.

The method combines two objectives. A latent-level contrastive task aligns each residue's language-model representation with the corresponding residue embedding from GearNet, pretrained on the AlphaFold database, injecting inter-protein structural signal. A physical-level task then trains the model to predict Foldseek 3Di structure tokens for each position, injecting intra-protein geometry. Because deposited structures vary widely in quality, a residue loss selection module uses a small reference model trained only on high-resolution entries to pick which per-residue losses are reliable enough to learn from.

The work comes from Mila – Quebec AI Institute and the Chandar Research Lab, with Yoshua Bengio and collaborators at Amgen, and was posted as a preprint in May 2025. AMPLIFY is the lab's own protein language model, so this checkpoint is also the recipe applied to a backbone its authors know intimately.

#Key Features

• Dual alignment objective: Contrastive latent alignment supplies structural context across proteins, while structure-token prediction supplies geometry within a protein; the two losses are weighted to sum to one.
• Quality-aware loss selection: A reference model trained on structures below 2.0 Å resolution with R-free under 0.20 identifies residues whose losses are informative rather than artifacts of poor experimental data, selecting the top 80% by excess loss.
• Drop-in checkpoints: The base tokenizer, masked-language-model head, and Hugging Face interface are preserved, so existing pipelines need no code changes.
• Compact released sizes: At 120M and 350M parameters, the checkpoints fit inference budgets where a 650M-parameter model would not.
• Evidence the method generalizes: Because this backbone is unrelated to ESM-2, gains here indicate the alignment recipe transfers across model families rather than exploiting one architecture.

#Technical Details

Structure alignment is trained on 129,732 PDB proteins (116,713 for training, 13,019 held out), using Foldseek's 20-token 3Di alphabet as the structure vocabulary. Released sizes are 120M and 350M parameters.

On supervised contact prediction, P@L/5 rises from 0.253 to 0.320 on trRosetta and from 0.155 to 0.169 on a withheld CASP16 set. The trRosetta gain is substantial; the CASP16 gain is slim, and notably smaller in both absolute and relative terms than the 0.181-to-0.288 jump the ESM-2 backbone shows on the same set. The honest reading is that the recipe transfers across families but not uniformly, and that CASP16 — the harder, more recent target — is where the difference between backbones shows up.

The authors also report that alignment raises pseudo-perplexity on a natural-sequence validation set, meaning some pure sequence-modeling capability is traded for the structural gain, though the flagship checkpoints stay competitive.

#Applications

The checkpoints suit anywhere an AMPLIFY model is already deployed and structural realism matters: zero-shot ranking of variants in deep mutational scanning and protein engineering campaigns, residue-residue contact and secondary structure prediction, thermostability and metal-ion-binding property prediction, and use as frozen feature extractors. The 120M and 350M sizes make them a reasonable choice where inference cost rather than peak accuracy is the binding constraint. Because the alignment is a lightweight post-training step rather than a new pretraining run, the same recipe can in principle be applied to other sequence models as they are released.

#Impact

The value of this checkpoint is partly as evidence rather than as a tool. A recipe demonstrated on a single backbone invites the objection that it exploits some quirk of that model; demonstrating it on a second, independently developed family answers that, and the uneven size of the gains across the two is itself informative about where structural post-training helps most. Code lives in the AMPLIFY repository under an MIT license and weights are on Hugging Face, though the checkpoints ship without substantive model cards, so the preprint remains the reference documentation. Evaluation is entirely computational, and the work has not yet been peer reviewed.