Prot-LAMBDA

Masked-residue prediction teaches a protein language model which positions co-vary, but it never tells the model that two residues 8 Å apart in space are related in a way that two residues 80 Å apart are not. The consequence is measurable: across the CASP14 and CAMEO benchmarks, the Spearman correlation between the distance separating a pair of ESM-2 embeddings and the Cα distance separating the residues themselves is about -0.01 — essentially none. Structure is present in these representations, but only as an incidental byproduct of evolutionary signal.

Prot-LAMBDA (Protein LAnguage Model Boosted with Distance Awareness), from the Kihara Lab at Purdue University, makes that correspondence an explicit training objective. Rather than regressing Cartesian coordinates, which is ill-posed without a canonical reference frame, the model builds a pair representation from the element-wise absolute difference between residue embeddings and trains it to predict whether each residue pair is in contact. Distance supervision therefore acts on the geometry of the embedding space itself, and the same correlation rises to 0.41.

The design contrasts with earlier structure-aware language models along a specific axis: SaProt feeds 3Di structural tokens in as input, S-PLM contrasts whole proteins by contact-map similarity, and ISM distills structure tokens into the encoder. Prot-LAMBDA instead constrains residue-level pairwise relationships directly, which is what lets a frozen copy of it substitute for a much larger encoder inside a downstream folding stack. The same preprint introduces LambdaFold, a compact structure prediction pipeline built on these representations.

#Key Features

• Distance-supervised pair representation: Pairwise absolute differences between residue embeddings are projected to 512 dimensions and trained against binary Cβ contacts at an 8 Å threshold, jointly with the masked language modeling loss.
• Axial attention with rotary embeddings: A single axial attention layer with 8 heads factorizes pairwise attention into row-wise and column-wise passes over the residue-residue matrix, with rotary positional embeddings applied along both axes so short- and long-range separations are encoded distinctly.
• Curriculum against catastrophic forgetting: Contact supervision starts on a random 25% of residue pairs and expands to the full interaction set over 40 epochs, preserving the pretrained sequence knowledge that joint training would otherwise erode.
• Strictly non-redundant training data: Every training sequence sharing more than 25% identity with a CASP14 or CAMEO target was removed, so benchmark numbers reflect generalization rather than memorization of test structures.
• Frozen drop-in encoder: Residue embeddings and axial attention maps project into the sequence and pair inputs of an AlphaFold 2 Invariant Point Attention module, making the model interchangeable with an ESM-2 backbone in a folding pipeline.

#Technical Details

Prot-LAMBDA initializes from ESM2-650M (33 transformer layers, embedding dimension 1280, pretrained on UniRef50) and fine-tunes only the top 11 layers, freezing the rest; low-rank adaptation performed worse in preliminary trials. Training used ~409,000 structures — 98,908 PDB chains released before January 2020 and 310,267 AlphaFold DB models drawn from UniClust30 clusters with more than 30 members — mixed 1:2 and cropped to fixed 256-residue windows for 80 epochs with AdamW at an effective batch size of 1,024, on four NVIDIA A100 GPUs for roughly 2,000 GPU-hours. That corpus is deliberately small next to the ~14 million proteins behind ESMFold.

Against ESM2-3B, a model roughly five times larger, Prot-LAMBDA raises long-range contact recall from 0.19 to 0.39 on CASP14 and from 0.27 to 0.60 on CAMEO, cuts psi-angle mean absolute error by 11.74% under linear probing on the PEER benchmark, and improves TM-score by 5.0% on CASP14 and 7.0% on CAMEO when both encoders are frozen behind the same IPA structure module. On TAPE contact prediction it exceeds SaProt and ISM, and on the CASP14-FM set it doubles the long-range precision@L/10 of the dedicated predictor SPOT-Contact-LM, 35.35% against 18.92%.

#Applications

The model serves as a sequence encoder wherever residue-level structural information is wanted without running a structure predictor: contact map and distogram prediction, secondary structure, backbone dihedral angles, relative solvent accessibility, and fold classification all work from a linear probe over frozen embeddings. Because it needs no multiple sequence alignment, it suits orphan proteins, de novo designs, and rapidly evolving viral sequences where homology search returns too little. The repository ships Jupyter notebooks for contact maps, distograms, structure prediction, and template-guided refinement, each running in under ten minutes on a single sequence.

#Impact

Prot-LAMBDA is a preprint awaiting peer review. Its central result — that a 650M-parameter encoder with explicit distance supervision beats a 3B-parameter one trained on sequence alone across five structure-related tasks — argues that architectural inductive bias can substitute for scale in structural representation learning, and the authors' insistence on 25% identity non-redundancy makes that claim harder to attribute to test-set leakage. The evaluation is entirely computational, on CASP14, CAMEO, TAPE, and PEER, with no experimental validation. Code and weights are released under GPL-3.0, and the authors invite anyone wanting alternative licensing terms, commercial use included, to contact them directly.