Loading…
Protein language model adding latent N-gram memory and sparse expert routing to an ESM-2 150M backbone for mutation-effect prediction.
No providers recorded yet. Browse all providers
Whether a substitution breaks a protein usually depends on what sits immediately around it — a conserved catalytic residue, an N-linked glycosylation sequon, a short motif that recurs across evolutionarily unrelated proteins. A dense protein language model represents those environments implicitly, spread across the same feed-forward parameters every residue passes through. It has no addressable place to store a recurring three-residue context and reuse it when that context reappears, and no way to spend extra computation where context matters most. The usual answer is a bigger backbone, which is expensive and saturates past mid scale.
ProtLingo takes the other route: keep the backbone small and add two explicit conditioning mechanisms. The first is a centered latent N-gram memory. Each residue representation is projected into route logits, binarized into route-specific discrete codes, and a centered three-residue window of those codes composes an address into a learned memory table; the retrieved vector is gated against the residue's own representation, refined by a depthwise convolution, and injected as a residual into selected layers. The second upcycles dense feed-forward blocks into sparse Mixture-of-Experts layers with one always-active shared expert and four routed experts, a top-1 router picking one per residue. Both paths start out reproducing the pretrained function exactly — memory tables at zero, routed experts carved from the pretrained weights — so continued pretraining departs from ESM-2 gradually rather than overwriting it.
ProtLingo comes from seven researchers at ShanghaiTech University and was posted to arXiv in September 2026. It is a preprint awaiting peer review; the authors state that source code and checkpoints will be released upon publication, and neither is available yet.
The backbone is the official ESM-2 150M checkpoint — 30 transformer blocks, hidden size 640, 20 attention heads, feed-forward width 2,560, a 1,024-token context, and ESM-2's tokenizer, vocabulary and masked-language-model head. LNgram modules sit before blocks 1, 11 and 21, each with a three-token window, two bits per route, memory dimension 8 and latent dimension 256, giving 320 routes per module and 125 addresses per route. Blocks 3–29 become MoE layers whose 2,560 channels split into two 1,280-channel halves, one seeding the shared expert and the other the four routed ones, holding the active feed-forward width at 2,560 and the model at 153M active parameters. Continued pretraining ran 40,000 steps at 786,432 tokens per step on a single eight-GPU node, over UniRef sampled 80% from UniRef50 and 20% from UniRef90, with 15% masking.
On the ProteinGym v1.3 DMS substitution benchmark — 217 assays and about 2.4 million measured variants, scored zero-shot by masked marginals — ProtLingo reaches 0.399 Spearman and 0.720 AUC, against 0.388 and 0.715 for ESM2-150M and 0.403 and 0.723 for ESM2-650M. Frozen-embedding ridge probes on FLIP give 0.544 on GB1, 0.576 on AAV and 0.650 on Meltome, ahead of ESM2-150M on the first two and level on the third. Supervised long-range contact prediction on 21 CASP15 domains yields P@L 0.485, P@L/2 0.657 and P@L/5 0.816 — matching the 150M backbone at the first two and trailing it slightly at the third, with ESM2-650M ahead on all three. Ablations show the memory module contributes more than the MoE layers on both DMS correlation and perplexity.
ProtLingo targets the mutation-effect workflows deep mutational scanning produces: ranking thousands of substitutions for stability, binding or activity without fitting an assay-specific head, and supplying frozen per-protein embeddings for supervised fitness probes on new landscapes. Because the active parameter budget stays near the 150M scale and training fits on one node, it suits protein-engineering groups that cannot run or adapt a 650M-parameter encoder.
The contribution is architectural rather than a leaderboard result: addressable local-context memory and residue-dependent routing can be grafted onto an existing dense checkpoint through continued pretraining, and the experts and memory routes then organize along recognizable biology — Pfam clans, Swiss-Prot functional sites, secondary-structure classes. The authors are explicit that the gains over ESM2-150M are bounded by the lightweight modules they chose. With no code or weights released, the near-term value is as a design for cheap conditional capacity rather than a checkpoint others can build on.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.