MSA Pairformer

Protein language modeling has largely been a scaling story: bigger single-sequence models trained on more sequences, with evolutionary information compressed into parameters. MSA Pairformer takes the opposite route. Developed by Yo Akiyama and colleagues in Sergey Ovchinnikov's lab at MIT, with Milot Mirdita and Martin Steinegger at Seoul National University, it reads evolutionary signal directly from a multiple sequence alignment (MSA) at inference time rather than memorizing it during training. The result is a 111M-parameter model that outperforms ESM-2 at 15 billion parameters on the task those models are usually judged by.

Earlier MSA-based models, notably MSA Transformer, were parameter-efficient but handicapped in practice: memory costs capped how many homologs they could read, and averaging across the alignment washed out signal specific to the query sequence. MSA Pairformer addresses both. A pairwise representation is updated bidirectionally against the MSA representation, and a query-biased outer product weights each aligned sequence by its relevance to the query, so subfamily-specific coevolution survives instead of being averaged into the family consensus.

The model was released as a preprint in August 2025 and published in Cell in 2026. A single pretrained checkpoint serves unsupervised contact prediction, protein-protein interface prediction, and zero-shot variant effect prediction with no task-specific retraining.

#Key Features

• Parameter efficiency over scale: At 111M parameters, it reaches an average long-range contact precision at L (P@L) of 0.52 on CASP15 targets, six points above ESM2-15B's 0.46 and eight above MSA Transformer's 0.44, with more than a 100-fold reduction in parameter count.
• Protein-protein interface contacts: On 25 evolutionarily conserved complexes it reaches a median precision at K of 0.53, versus 0.29 for MSA Transformer and 0.01 for both ESM2-15B and gLM2 — roughly a two-fold improvement over the strongest prior method.
• Query-biased sequence weighting: Sequences are weighted by relevance to the query rather than uniformly, recovering subfamily-specific homodimer interface contacts that uniform weighting averages away, even when the subfamily is a small minority of the alignment.
• No contact-variant trade-off: Single-sequence models improve at contacts but degrade at variant effect prediction as they scale; this model achieves an average Spearman correlation of 0.47 across the 219 ProteinGym deep mutational scanning substitution assays while holding the contact lead.
• Calibrated, non-hallucinating predictions: Ablations show triangle multiplicative updates suppress indirect correlations, and unlike MSA Transformer the model does not invent contacts when covariance is removed, making it usable for screening candidate interacting sequence pairs.

#Technical Details

The architecture stacks 22 layers, each applying pair-weighted averaging, a query-biased outer product built on pre-softmax differential attention, outgoing and incoming triangle multiplicative updates, and transitions. Training used masked amino acid prediction over the 270,000 Uniclust30 alignments in OpenProteinSet, in two phases: pretraining with uniform sequence weighting, then attention finetuning that swaps the outer product mean for the query-biased variant. Contacts are read out by a logistic regression head over the 256-dimensional pair representation, fit on the same 20 trRosetta training samples used for the MSA Transformer and ESM-family contact heads. The model handles alignments of up to 4,096 sequences and a context window of 896 residues in the ProteinGym evaluation, and it does not require average product correction — a background-subtraction step both MSA Transformer and the ESM-2 family depend on. High-confidence calls are trustworthy: of 478 predicted interface contacts with probability above 0.8, 85% fall within 8 Å and 95% within 12 Å in the crystallographic structure.

#Applications

The model suits any workflow where coevolutionary contacts are the endpoint rather than a step toward a folded structure: mapping interaction interfaces in complexes with no solved structure, screening which paralogs in a family actually interact, and ranking missense variants by predicted fitness effect. Because it runs on modest hardware, it lowers the barrier for labs that cannot host multi-billion-parameter checkpoints, and its interface performance addresses a regime where single-sequence models such as gLM2 largely fail. A Colab notebook with automated MSA generation via an updated ColabFold MMseqs2 protocol makes the model usable without local installation.

#Impact

MSA Pairformer is an argument that the scaling trajectory of protein language models is not the only path to better representations. Retrieving evolutionary context at inference lets the model improve automatically as sequence databases grow, without retraining a larger network, and it sidesteps the trade-off where contact accuracy and variant effect accuracy pull in opposite directions as single-sequence models scale. Code and weights are released under a permissive modified MIT license that allows academic and commercial use. The main constraint is structural: predictions require an MSA, so query sequences with few detectable homologs — de novo designs, orphan proteins, fast-evolving families — fall outside the regime where the approach is strongest, and sequence length is bounded by the memory available for the pairwise representation.