Loading…
Protein language model embedding enrichment that injects structural and dynamical signal as a low-energy residual, using sequence alone at inference.
No providers recorded yet. Browse all providers
Two proteins can share a fold while sharing almost no sequence identity, and a model trained only on amino acid sequence has no direct way to see it. The evidence that would settle the question — the arrangement of the backbone in space, and how much of it moves — is precisely what a sequence encoder never observes. The usual remedy is to fuse structural features into the embedding, but fusion rewrites the geometry that evolutionary signal had organized, and it demands a structure at inference time that most proteins do not have.
ProtEnrich, from Gabriel Bianchin de Oliveira and Fahad Saeed at Florida International University, avoids the fusion entirely. It decomposes a frozen sequence embedding into two learned subspaces: an anchor subspace trained to reconstruct the original embedding and preserve its semantics, and an alignment subspace trained contrastively to sit close to structural and dynamical embeddings of the same protein. The two are recombined as an anchor plus a scaled alignment term, with the scaling coefficient fixed at 0.039, so biophysical information enters as a deliberately small residual rather than a competing signal. An energy penalty on the alignment norm enforces that: removing it pushes the alignment-to-anchor norm ratio from 0.06 to 0.98, at which point the residual stops being a residual.
The teachers are consulted only during pretraining: ProstT5 supplies a 1024-dimensional encoding over the Foldseek 3Di alphabet, RocketSHP a 20-dimensional descriptor of conformational flexibility. At inference ProtEnrich sees a sequence embedding and nothing else. ISM reaches the same destination by distilling structure tokens into an ESM2 backbone during pretraining; ProtEnrich leaves every backbone untouched, which is why it ships for eight encoders rather than one.
Pretraining used 550,120 SwissProt proteins with AlphaFold DB structures, run for 30 epochs on a 95/5 train-validation split, separately for each of the eight backbones. All four representations — anchor, alignment, structure, dynamics — live in a shared 1024-dimensional latent space and are projected to 256 dimensions for an InfoNCE contrastive objective with a learnable temperature initialized at 0.1. The loss sums contrastive alignment and energy regularization at weight 1.0 each, structural and dynamical reconstruction at 0.5, and sequence reconstruction at 0.1. Backbones stay frozen and are mean-pooled over amino acid tokens, sequences truncated at 1022 residues. The released checkpoints hold 13.4M to 18.2M parameters, depending on the backbone's embedding width.
Results are deltas against the unenriched backbone, not standalone scores. On the TAPE remote homology benchmark the largest gains went to ProtBERT, whose family-level Precision@10 rose from 0.369 to 0.490 and MRR from 0.733 to 0.846. ProGen2's fluorescence RMSE fell from 1.033 to 0.865, roughly a 16% error reduction, and ESM2 T36's metal-ion-binding AUCROC rose from 0.809 to 0.833 — ahead of ISM's 0.821 on the same backbone. The gains are selective: solubility and multiclass localization were flat or worse across most backbones, and the two encoder-decoder models gained least on structural retrieval.
The practical target is any pipeline already consuming frozen protein language model embeddings that cares about properties the fold determines: remote homology search where sequence identity is too low for alignment methods, metal ion binding and other ligand-interaction prediction, and fluorescence or stability regression on engineered variants. Because enrichment is a module rather than a new backbone, adopting it costs one forward pass through a sub-20M-parameter network, and the imputed structural and dynamical vectors are usable on their own for proteins with no experimental structure.
ProtEnrich is a preprint awaiting peer review, and its checkpoints, code and benchmark data carry a non-commercial license that limits industrial uptake. Its contribution is a design principle rather than a leaderboard position: multimodal knowledge can be added to a representation as a norm-constrained residual in an orthogonal subspace, leaving the original geometry measurably intact, instead of being fused into it. That framing is what lets one method be instantiated across eight unrelated architectures without touching any of them, and it answers a recurring problem in protein representation learning: the structural information a model needs is usually unavailable at the moment it needs it.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.