LC-SEPLM

Protein language models such as ESM-2 learn from amino-acid sequences alone, recovering masked residues from surrounding context. That objective supervises residue identity, not the spatial relationships that appear once a chain folds. LC-SEPLM (Long-range Contact-supervised ESM Protein Language Model) attacks the gap directly: an auxiliary training signal asks the model to predict whether two residues far apart in the sequence end up touching in three dimensions, and the structural machinery is then dropped at inference.

The model was developed by Chen Wang, Boming Kang, and Qinghua Cui across Peking University, Hubei University of Technology, and Wuhan Sports University, and released as an arXiv preprint. It sits between purely sequence-based language models and joint sequence-structure encoders such as ESM-GearNet, which fuse a language model with a geometric graph network and therefore need coordinates at deployment. LC-SEPLM's claim is that structural information can be injected during adaptation instead of consumed during inference, leaving the deployment interface unchanged.

The granularity of that supervision is the substantive design choice. Rather than regressing inter-residue distances or full contact maps, the model trains on binary long-range contact labels for sampled residue pairs — labels that keep the identity of both residues and state plainly whether two sequence-distant positions become spatially connected.

#Key Features

• Long-range contact supervision: A positive label requires a sequence separation of at least 12 residues and a Cα–Cα distance of at most 8 Å, concentrating the signal on fold-defining contacts that sequence context captures only indirectly.
• Sequence-only inference: Contact prediction is a training-time auxiliary task; downstream tasks consume embeddings computed from sequence alone, with no coordinates, MSAs, or predicted structures required.
• Pair-specific cross-attention queries: Each residue pair forms a query from the two residue states, their difference, their elementwise product, and their sequence separation, which then cross-attends over the whole sequence — so contact prediction draws on global rather than local context.
• Parameter-efficient adaptation: LoRA adapters of rank 4 target only the query and value projections in the last eight transformer layers, leaving the pretrained backbone frozen.
• Retained language-modeling loss: A masked-language-modeling term is kept alongside the contact loss at weight 0.6, guarding the pretrained representation against drift during adaptation.

#Technical Details

The backbone is the 150-million-parameter ESM-2 checkpoint esm2_t30_150M_UR50D: 30 transformer layers with 640-dimensional hidden states. LoRA adapters (rank 4, α = 8, dropout 0.05) target the q_proj and v_proj matrices of layers 22–29. For a residue pair, concatenating both hidden states, their absolute difference, their Hadamard product, and the sequence offset yields a 2,561-dimensional vector; an MLP projects it back to the model width to serve as the query in a cross-attention block whose keys and values come from the full sequence, and a sigmoid-activated MLP head emits the contact probability. Adaptation ran for 14 epochs on 500,000 AlphaFold Swiss-Prot structures, with 2,000 proteins each held out for validation and test, sequences truncated to 512 residues, and up to 768 pairs sampled per protein at roughly a 2:1 negative-to-positive ratio.

All eight protein-level tasks improved over the matched ESM-2 baseline. The gain concentrates in Pfam-clan remote-homology recognition, where macro-F1 rises from 0.6122 to 0.6769; the rest are modest, such as GO molecular function at 0.7096 to 0.7195 and Pfam family classification at 0.9787 to 0.9843. Against ESM-S on its own EC benchmark, LC-SEPLM scores 0.4652 versus 0.2880 at the 95% sequence-identity cutoff (+0.1771), with comparable margins at the 50% and 30% cutoffs. Two tasks regress slightly: ProteinGym mutation-effect prediction (−0.0076) and PTM-site prediction (−0.0001).

#Applications

LC-SEPLM is a drop-in replacement for an ESM-2 encoder in annotation pipelines that only ever see sequences: enzyme-class assignment, Gene Ontology term prediction, subcellular localization, membrane-protein classification, and family or clan assignment for proteins with no close characterized relative. The remote-homology result is the most practically useful — placing a sequence in the right Pfam clan after family-level similarity has decayed is precisely where similarity search fails, a common situation in metagenomic and orphan-gene annotation. Because inference needs no structure, the encoder can be swapped in without changing surrounding infrastructure.

#Impact

The contribution is a clean demonstration that structural supervision need not become a deployment dependency: a small LoRA adaptation with a residue-pair contact objective shifts global protein representations while leaving the sequence-only interface intact. The evidence is honestly bounded. Gains outside remote homology are fractions of a percentage point, local-residue tasks do not benefit, and the ESM-S comparison uses published scores rather than a retrained matched pipeline. Training on predicted rather than experimental structures also risks importing the structure predictor's biases into the contact labels. As a preprint not yet peer reviewed, with source code and model files stated as forthcoming upon publication, independent reproduction and benchmarking remain ahead of it.