Loading…
Protein language model pairing sequence with quantized local-structure tokens via disentangled attention, for zero-shot variant effect prediction.
No providers recorded yet. Browse all providers
A catalytic pocket and an unremarkable surface loop can end up carrying the same structural token when the alphabet describing them looks only at a residue's immediate sequence neighbors. That is the resolution limit of Foldseek's 3Di alphabet, the bridge from coordinates to tokens that structure-aware protein language models had adopted: fast and good at fold-level similarity, but it blurs exactly the micro-environments — active sites, binding pockets — where function is decided.
ProSST replaces that alphabet with one learned at higher resolution. Each residue is described by a star-shaped graph over its 40 nearest neighbors in space, encoded by a geometric vector perceptron trained with a denoising objective, and then quantized to the nearest centroid of a k-means codebook. The resulting integer sequence runs alongside the amino-acid sequence into a transformer whose self-attention is replaced by sequence-structure disentangled attention, so residue-to-structure and structure-to-residue interactions are scored by their own projection matrices rather than summed into one embedding before attention sees them.
The model comes from Shanghai Jiao Tong University with the Shanghai Artificial Intelligence Laboratory, East China University of Science and Technology, the Chinese University of Hong Kong, and SJTU's Chongqing AI institute, and was published at NeurIPS 2024. It sits beside SaProt, which builds its structure-aware vocabulary from Foldseek tokens, and one step past sequence-only models such as ESM-2.
The backbone is a 12-layer transformer with 12 attention heads, 768 hidden dimensions and a 3072-dimension feed-forward block, totalling roughly 117 million parameters in the released ProSST-2048 checkpoint. It was pretrained with a 15% masked-language-model objective on 18.8 million AlphaFold DB structures — the 90% sequence-identity reduced set — for 500,000 steps on eight A800 GPUs. The structure encoder and codebook were fit separately on CATH43-S40: 31,270 domains for the denoising autoencoder, and 4,735,677 extracted local structures for k-means.
On ProteinGym's 217 deep mutational scanning assays, ProSST reaches Spearman ρ = 0.504, against 0.458 for SaProt-650M and 0.414 for ESM-2 650M, and ρ = 0.663 on the thermostability subset. Fine-tuned, it reaches 94.68% on DeepLoc subcellular localization and ρ = 0.724 on thermostability regression. The ablation is the informative number: with the structure vocabulary removed the same model falls to ρ = 0.392, in line with sequence-only baselines.
The primary use is directed evolution. Given a wild-type structure, ProSST ranks point and multi-point mutants before any are made, narrowing a library to the variants worth expressing and assaying. The same embeddings fine-tune for subcellular localization, metal ion binding and GO annotation, and the quantizer works on its own as a structure tokenizer for other models.
ProSST established that the structure alphabet, not just the sequence backbone, is a lever worth pulling: a 117M-parameter model matching 650M sequence-structure hybrids traces directly to a finer-grained tokenizer. It is now a standard comparison point in variant-effect work.
The same group showed how much further the released checkpoint goes without retraining. ProtREM — renamed VenusREM in the published version, so the literature carries both names — loads ProSST-2048 unchanged and adds a training-free evolutionary term: per-column amino-acid frequencies counted over a Jackhmmer/EVCouplings homolog alignment, log-softmaxed and mixed with the model's own log-probabilities as (1 − α)·ProSST + α·MSA at α = 0.8, so α = 0 recovers plain ProSST. That one addition raises the average Spearman ρ over ProteinGym's 217 substitution assays from 0.504 to 0.518, first on the substitution leaderboard in April 2025. In the wet lab, 8 of 10 mutants it designed on a thermostable phi29 DNA polymerase gave over 3-fold amplification gains at 42 °C — Y449G reached 6.5-fold — and 3 of the 8 also raised the melting temperature. It was published in Bioinformatics for ISMB/ECCB 2025.
Three constraints bound its use: local-structure encoding is computationally heavy relative to sequence-only inference; every input needs a structure, experimental or predicted; and the retrieval-enhanced configuration additionally needs a homolog alignment, so its gain is smallest on sequences with few detectable homologs. The code, ProSST's and VenusREM's alike, is released under CC BY-NC-ND 4.0, which rules out commercial deployment and derivative works, and the weight repositories declare no license of their own.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.