Loading…
Homology-aware protein language model on a recurrent xLSTM backbone, generating and scoring sequences from long contexts of unaligned homologs.
No providers recorded yet. Browse all providers
A protein's evolutionary neighbours carry most of the information about which residues matter, and the bluntest way to hand a model that information is to paste hundreds of homologous sequences into one context and let it read them all. Done honestly, that context runs past a hundred thousand tokens — and self-attention charges quadratically for every one of them, then keeps a key–value cache that grows with each token generated. The context length that makes homology useful is precisely the regime where transformers become impractical.
Prot-xLSTM answers that with a recurrent backbone. It comes from the ELLIS Unit Linz and LIT AI Lab at Johannes Kepler University Linz, together with NXAI, and is the protein member of Bio-xLSTM, a report that tailors the xLSTM architecture to biological and chemical sequences and also introduces DNA-xLSTM for genomes and Chem-xLSTM for SMILES strings. xLSTM revives the LSTM with exponential gating and a matrix memory cell; its mLSTM block parallelises across the sequence during training like attention, but at inference it decodes recurrently in constant memory with runtime linear in sequence length. Long homolog contexts stop being a scaling problem and become an ordinary one.
The training recipe follows ProtMamba: unaligned homologous sequences are concatenated into a single long example and trained with causal language modelling under a fill-in-the-middle objective, so a single checkpoint can extend a sequence, inpaint a masked span, or score a substitution. Nothing about the pipeline requires a multiple sequence alignment, which sidesteps alignment construction and its failure modes on insertions, repeats, and disordered regions.
Two configurations were trained, both stacking 16 mLSTM blocks with a 2:1 projection ratio and rotary position embeddings: Prot-xLSTM-26M at embedding dimension 512 and Prot-xLSTM-102M at 1,024. Training data is the filtered OpenProteinSet — 270,000 UniClust30 clusters covering 508 million sequences and 110 billion residues. The released checkpoints cover 30 billion tokens for the 26M model and 60 billion for the 102M model. At a context of 2^17 tokens, Prot-xLSTM-102M reaches a test perplexity of 6.83 against 7.47 for ProtMamba-107M, despite seeing under a third of ProtMamba's training tokens. On the ProteinGym zero-shot DMS substitution benchmark of 217 assays it averages Spearman ρ = 0.416, ahead of ESM-2 at 15B parameters (0.401) and ProGen2 XL at 6B (0.391); models that consume an MSA directly, such as TranceptEVE L (0.456), or structure tokens, such as SaProt (0.457) and ProSST (0.507), remain ahead. For homology-conditioned generation, 2,500 sequences were sampled for each of 19 clusters and compared to natural proteins by Kolmogorov–Smirnov statistic; Prot-xLSTM's distributions were closest to natural on three of five metrics, including HMMER score, pLDDT, and pTM.
The practical uses are family-conditioned protein design — prompt with a cluster's homologs and sample new members that fold and score like natural ones — and infilling, where a functional motif is held fixed and the surrounding sequence regenerated. Zero-shot fitness prediction makes it a cheap first pass for prioritising mutants in a directed-evolution campaign, and because scoring needs neither an alignment nor a predicted structure it applies to sequences where both are hard to obtain. Two example notebooks in the repository run generation and variant scoring directly from the released checkpoints.
Prot-xLSTM's contribution is architectural: it shows that a recurrent model with matrix memory is a viable protein language model backbone, competitive with the state space models and transformers that dominate the field, and that its efficiency advantage is largest exactly where homology conditioning needs it. The code is Apache-2.0 licensed and the checkpoints are downloadable, though they are hosted on institutional web space rather than a model hub and carry no separate licence of their own. The models are research-scale — 102M parameters against the billions used by single-sequence models it outscores — and the authors note that performance beyond the billion-parameter regime is untested, along with training-set biases inherited from the underlying sequence databases.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.