LC-PLM

Almost every protein language model is a transformer trained on short windows. ESM-2 uses a 1,024-token context because that covers 97.4% of individual proteins — but the proteins that fall outside it are large multidomain chains, and the biology that matters most often involves complexes, not single sequences. Attention makes the obvious fix expensive: cost grows quadratically with context length during training, so simply widening the window is not affordable at pretraining scale.

LC-PLM, from researchers at Amazon and Cornell University, replaces attention with a selective structured state space model. The backbone is BiMamba-S, a bidirectional Mamba variant in which the forward and backward passes share their input and output projection layers. Sharing those projections is what makes the design distinctive: at a fixed parameter budget it buys twice the depth of a naive bidirectional Mamba, which the authors show improves evaluation loss by 4.5% and structure-prediction accuracy by up to 6.7%. Training is standard masked language modeling — 15% of amino acid tokens masked, with the usual 80/10/10 replacement scheme — on UniRef50 and UniRef90.

A second variant, LC-PLM-G, adds a graph-contextual training stage. Random walks over a protein–protein interaction graph are linearized into token sequences with special graph-identifier tokens, so a single long context spans several interacting proteins at once. This is where the long-context argument becomes biological rather than merely architectural: the model can condition one protein's representation on its partners.

#Key Features

• Linear-cost long context: The state space backbone scales linearly rather than quadratically in sequence length and carries no fixed positional encoding, so a model pretrained at a 1,024-token block size still runs cleanly over multidomain proteins and complexes that transformer protein LMs truncate.
• Depth from shared projections: BiMamba-S ties the forward and reverse projection layers, allowing roughly twice the layer count at equal parameter count — the released checkpoint is 48 layers at a 1,536-dimensional hidden state.
• Genuine length extrapolation: Trained only on the 128–256 residue bin, LC-PLM holds low evaluation loss across all bins up to 4,096–8,192, where ESM-2 with rotary embeddings degrades on both shorter and longer sequences.
• Graph-contextual variant: LC-PLM-G contextualizes PPI graphs in a second training stage; its embeddings recover Louvain community structure that plain LC-PLM embeddings do not.
• Favorable scaling behavior: Across 110M to 1.3B parameters, LC-PLM shows better neural scaling than size-matched ESM-2, averaging 13.5% lower evaluation loss.

#Technical Details

The released 740M-parameter checkpoint uses 48 bidirectional Mamba layers with d_model 1,536 and a 128-token vocabulary, loaded through transformers with the ESM-2 tokenizer. Pretraining ran to 100B and 1T tokens from UniRef50, with held-out UniRef90 sequences for evaluation. On structure prediction through LMFold, a lightweight folding head trained on 7,872 OpenFold chains with FAPE and distogram losses, LC-PLM-740M at 100B tokens reaches TM-scores of 0.5012 on CASP15-multimers, 0.4014 on CASP14, and 0.6128 on Benchmark2, against 0.3992, 0.3403, and 0.4724 for ESM-2-650M at the same token budget — improvements of 20.8%, 17.6%, and 29.5%. Trained to 1T tokens it reaches 0.5515, 0.4650, and 0.7075. On TAPE it scores 35.14 on remote homology and 85.07 on secondary structure, versus 26.57 and 79.86 for equally trained ESM-2, and it exceeds the public 1T-token ESM-2-650M on both; ProtMamba and CARP trail further behind. Zero-shot fitness prediction across the 217 deep mutational scanning datasets in ProteinGym gives Spearman ρ = 0.378 at 100B tokens against 0.295 for ESM-2-650M. Public ESM-2 retains an edge on contact-map prediction.

#Applications

LC-PLM is a general-purpose embedding model: residue-level and sequence-level representations feed structure prediction heads, function and stability predictors, remote homology search, and zero-shot variant scoring. The long context is the practical differentiator for anyone working on large multidomain proteins or on complexes, where a 1,024-token model must truncate or chunk. LC-PLM-G's graph stage suits interactome-centric problems — link prediction over PPI networks, or function prediction where a protein's neighbors are informative.

#Impact

The result that matters beyond this specific checkpoint is architectural: state space models are a viable protein language model backbone, and depth bought through shared projections is a better use of a fixed parameter budget than width. That reframes protein LM scaling, which has been dominated by transformer width and token count. Two limitations bound adoption. The work remains a preprint, posted to bioRxiv in October 2024 and arXiv in November 2024, with revisions through April 2025. And the release is licensed CC-BY-NC-4.0, which rules out commercial use — an unusual restriction for an industrial protein language model release, and one that will keep LC-PLM out of pipelines where permissively licensed alternatives are available. The 740M weights ship via Git LFS in the repository itself with a working transformers inference example; the graph-contextual LC-PLM-G weights are not part of that release.