ProtMamba

Protein language models split into two camps. Single-sequence models such as ESM-2 read one protein at a time and learn evolutionary constraint implicitly from a large corpus. Family-based models such as MSA Transformer read a multiple sequence alignment and get that constraint explicitly, which usually helps — but an MSA is a constructed artifact, expensive to build, sensitive to alignment errors, and awkward for sequences with insertions, repeats, or disorder.

ProtMamba, from Anne-Florence Bitbol's lab at EPFL and the SIB Swiss Institute of Bioinformatics, takes the homology information without the alignment. Training examples are formed by concatenating hundreds of unaligned homologous sequences into a single very long context, so the model sees the same evolutionary signal an MSA carries but never has to commit to a column assignment. Making that practical requires a long context at manageable cost, which is why the backbone is Mamba, a state space architecture whose memory scales linearly in sequence length rather than quadratically as attention does.

The training objective is fill-in-the-middle: parts of a sequence are masked and moved to the end, so the model learns both autoregressive continuation and infilling in a single pass. This makes one checkpoint useful for generation, for inpainting specific regions, and for scoring variants, without task-specific heads or fine-tuning. The work first appeared on bioRxiv in May 2024 and was published in Bioinformatics in 2025.

#Key Features

• Homology conditioning without MSAs: Context is a concatenation of unaligned homologs separated by delimiter tokens, sidestepping alignment construction entirely while retaining family-level evolutionary information.
• Fill-in-the-middle training: Combining autoregressive modeling with masked infilling in one objective yields a model that can extend a sequence, fill an arbitrary masked span to a specified length, or score a point mutation.
• Very long context: The Long variant was trained until the context reached 2^17 tokens — enough for hundreds of full-length homologs — with linear rather than quadratic memory scaling.
• Prompt-engineered conditioning: Restricting the context to a subfamily, or to experimentally active variants only, steers both generation and scoring, which the authors propose as an alternative to fine-tuning.
• Fast variant scoring: Context caching under the fill-in-the-middle objective means the first variant in a dataset costs about 1.17 s and every subsequent one about 0.76 s on average.

#Technical Details

ProtMamba has 107M parameters across 16 layers with embedding dimension 1,024; positional information is concatenated with token embeddings rather than added, with half the embedding dimension allocated to each. Training used a filtered subset of OpenProteinSet — 268,000 maximally diverse MSA clusters comprising 508 million sequences and 110 billion residues — on two GPUs, with 192 and 500 clusters held out for validation and testing. Four checkpoints are released: foundation and fill-in-the-middle-fine-tuned versions of both the standard model (trained to 2^15 tokens of context) and the Long model (2^17). On the 217-dataset ProteinGym benchmark, ProtMamba reaches Spearman ρ = 0.406 from sequence alone and 0.432 with retrieval, against 0.387 for ESM-2 150M, 0.414 for ESM-2 650M, 0.421 for MSA Transformer, and 0.447 for PoET at 201M parameters — while scoring all of ProteinGym in 7–10 minutes on a single RTX A6000 versus roughly 10 hours for PoET. Structure-aware models remain ahead: SaProt reaches 0.457 and ProSST 0.507. On chorismate mutase activity prediction, conditioning on active variants only lifts Spearman ρ to 0.53, above published DCA-energy (0.41) and logistic-regression (0.43) baselines.

#Applications

The natural uses are family-conditioned protein design — generating new members of a family by prompting with its homologs — and motif inpainting, where a fixed functional motif is retained and the surrounding sequence is regenerated to a chosen length. Zero-shot fitness prediction makes it a cheap first pass for prioritizing mutants in directed evolution, and its speed advantage matters when scoring large variant libraries. Because it does not depend on alignment, it also handles intrinsically disordered regions, which MSA-based methods struggle with.

#Impact

Code and weights are Apache-2.0 licensed, with checkpoints distributed through GitHub releases and the repository archived on Zenodo. ProtMamba's practical argument is efficiency: at 107M parameters and two GPUs of training it matches or beats models an order of magnitude larger in compute, which the authors attribute to long-context conditioning rather than scale. It does not lead ProteinGym — structure-aware models score higher, and PoET edges it among structure-agnostic ones — but it establishes state space models as a viable protein-language-model backbone and shows that unaligned homolog context is a workable substitute for the MSA.