AMPLIFY

Scaling a protein language model rests on an assumption about its corpus: that a public sequence database is a faithful sample of the fitness landscape nature explored. It is not. UniRef50 collapses each cluster of similar sequences to a single representative, discarding the redundancy that is itself evidence of which regions of sequence space are real and functional. The databases also carry a large population of hypothetical entries, some of which are later reannotated as non-coding transcripts and were never proteins at all. Train a bigger network longer on that mixture and much of the extra capacity goes into memorizing it.

AMPLIFY — the Amgen-Mila Protein Language model for InFerence and discoverY — is the answer its authors give to the title of their paper, "Protein Language Models: Is Scaling Necessary?". Instead of adding parameters, they rebuilt the training corpus around UniRef100 rather than UniRef50, added paired antibody sequences and structural domains, and modernized the encoder for throughput. The result is a masked language model released at 120M and 350M parameters that performs comparably to models two orders of magnitude larger. It was developed by Amgen with Mila – Quebec AI Institute and the Chandar Research Lab, and posted as a preprint in September 2024.

The architecture is deliberately close to ESM-2, which serves as both the starting point and the control: the authors retrained ESM2 models from scratch on their own data to separate the contribution of curation from the contribution of code. AMPLIFY is also the backbone for the lab's later structure-aware post-training work, SaAMPLIFY.

#Key Features

• Curation in place of scale: Training on UniRef100 with paired antibody and SCOP sequences, rather than a redundancy-reduced subset, improves likelihood and sequence recovery independent of model size or training duration.
• Two-stage context extension: Pre-training runs for 1M steps at 512 residues, then extends the context window to 2048 for a further 25K steps (120M) or 50K steps (350M), avoiding the cost of long-context training throughout.
• Efficiency-oriented encoder: SwiGLU activations, RMSNorm, rotary position embeddings, FlashAttention, bias-free projections, and 64-dimensional attention heads cut both training and inference cost.
• Non-protein discrimination: Embedding geometry separates intrinsically disordered human proteins from hypothetical database entries at 0.94 ROC-AUC, where AlphaFold2 pLDDT confidence scores 0.44.
• Complete open release: Codebase, training and validation sets, and all four checkpoints are published under an MIT license.

#Technical Details

The 120M model uses 24 layers with hidden size 640 and 10 attention heads; the 350M model uses 32 layers with hidden size 960 and 15 heads. The vocabulary is trimmed from 33 to 27 tokens and gains a | chain separator for multi-chain inputs such as antibody heavy and light chains. Training data is UR100P, assembled in December 2023 from UniRef100, the Observed Antibody Space, and SCOP 2, with ambiguous residues removed and MMseqs2 used to strip sequences above 90% identity to the held-out sets.

AMPLIFY 350M outperforms ESM2 15B on sequence recovery with 43 times fewer parameters, 17 times fewer training FLOPs (4.6x10^21 against 8.1x10^22), and 24 to 29 times higher inference throughput depending on sequence length. AMPLIFY 120M reaches parity with ESM2 3B at roughly a tenth of the compute. The study behind these numbers trained 21 models from scratch, totaling 1.68x10^22 FLOPs, and evaluated 18 further public checkpoints as controls. The advantage is not uniform: ESM-2 still leads on unsupervised contact prediction from attention maps, which the authors attribute to AMPLIFY's smaller number of attention heads, and the gap narrows substantially when a random forest replaces the usual logistic-regression probe.

#Applications

The checkpoints serve the standard protein language model roles — residue and sequence embeddings for downstream property prediction, masked-position scoring for mutation suggestion and consensus design, and antibody work where the paired OAS training data is directly relevant. The efficiency argument matters most for large in-silico screens, where a 350M model that matches a 15B one turns an infrastructure problem into a workstation job. The non-protein discrimination result also suggests a curation use: flagging database entries whose embeddings look unlike real proteins for expert reevaluation.

#Impact

AMPLIFY is a data-quality argument delivered as a model. Its central claim, that careful curation competes with scale, is supported by an unusually explicit set of controls, and the release of the pre-training codebase and datasets alongside the weights is what makes that claim checkable rather than merely asserted. That release has been taken up in practice: the same repository now hosts structure-aware derivatives of both AMPLIFY and ESM-2. The work remains a preprint, revised in January 2026 without change to its abstract, and the evaluation is entirely computational, resting on sequence recovery, perplexity, and embedding-space analyses rather than experimental validation.