ProSST

A catalytic pocket and an unremarkable surface loop can end up carrying the same structural token when the alphabet describing them looks only at a residue's immediate sequence neighbors. That is the resolution limit of Foldseek's 3Di alphabet, which structure-aware protein language models had adopted as their bridge from coordinates to tokens: it is fast and it captures fold-level similarity, but it blurs exactly the micro-environments — active sites, binding pockets — where function is decided.

ProSST replaces that alphabet with one learned at higher resolution. Each residue is described by a star-shaped graph over its 40 nearest neighbors in space, encoded by a geometric vector perceptron trained with a denoising objective, and then quantized to the nearest centroid of a k-means codebook. The resulting integer sequence runs alongside the amino-acid sequence into a transformer whose self-attention is replaced by sequence-structure disentangled attention, so residue-to-structure and structure-to-residue interactions are scored by their own projection matrices rather than being summed into one embedding before attention ever sees them.

The model comes from Shanghai Jiao Tong University with the Shanghai Artificial Intelligence Laboratory, East China University of Science and Technology, the Chinese University of Hong Kong, and SJTU's Chongqing AI institute, and was published at NeurIPS 2024. It sits directly beside SaProt, which builds its structure-aware vocabulary from Foldseek tokens, and one step past sequence-only models such as ESM-2.

#Key Features

• Learned local-structure quantizer: A GVP encoder embeds each residue together with its spatial neighborhood, and a pre-trained k-means model assigns the discrete token — capturing micro-environment detail that per-residue geometric alphabets discard.
• Sequence-structure disentangled attention: Of the nine attention terms a residue/structure/position decomposition yields, five are kept — the four that carry residue-relevant signal plus residue-to-residue — following the DeBERTa formulation.
• Structure-conditioned zero-shot scoring: Mutant effects are scored as the log-likelihood ratio of mutant to wild-type residue, conditioned on the wild-type structure token sequence, with no fitness data required.
• A family of structure vocabularies: Checkpoints exist at codebook sizes 20, 128, 512, 1024, 2048, and 4096, alongside Foldseek- and DSSP-tokenized variants that make the quantizer's contribution directly measurable.

#Technical Details

The backbone is a 12-layer transformer with 12 attention heads, 768 hidden dimensions and a 3072-dimension feed-forward block, totalling roughly 117 million parameters in the released ProSST-2048 checkpoint. It was pretrained with a 15% masked-language-model objective on 18.8 million AlphaFold DB structures — the 90% sequence-identity reduced set — for 500,000 steps in BF16 on eight A800 GPUs. The structure encoder and codebook were fit separately on CATH43-S40: 31,270 domains for the denoising autoencoder, and 4,735,677 extracted local structures for k-means.

On ProteinGym's 217 deep mutational scanning assays, ProSST reaches Spearman ρ = 0.504, against 0.458 for SaProt-650M and 0.414 for ESM-2 650M, and ρ = 0.663 on the thermostability subset. Fine-tuned, it reaches 94.68% on DeepLoc subcellular localization, 76.67% on metal ion binding, ρ = 0.724 on thermostability regression, and 0.501 F1-max on GO cellular component. The ablation is the informative number: with the structure vocabulary removed the same model falls to ρ = 0.392, in line with sequence-only baselines.

#Applications

The primary use is directed evolution. Given a wild-type structure — experimental or predicted — ProSST ranks point and multi-point mutants before any of them are made, narrowing a library to the variants worth expressing and assaying, which is where protein engineers spend their budget. The same embeddings fine-tune for subcellular localization, metal ion binding, thermostability and GO annotation, and the quantizer is usable on its own as a structure tokenizer for other models.

#Impact

ProSST established that the structure alphabet, not just the sequence backbone, is a lever worth pulling: a 117M-parameter model matching or beating 650M sequence-structure hybrids traces directly to a finer-grained tokenizer. It has become a standard comparison point in subsequent variant-effect work, and the group extended it into a retrieval-augmented successor and folded it into their protein-engineering platform. Two constraints bound its use: the local-structure construction and encoding is computationally heavy relative to sequence-only inference, and every input needs a structure, which restricts it to proteins where one can be obtained. The code is released under CC BY-NC-ND 4.0, which rules out commercial deployment and derivative works, and the weight repositories declare no license of their own.