CCPL

A protein structure encoder has no equivalent of masked language modeling. The trick that made sequence models work — corrupt a token, predict it back — has no safe analogue in three dimensions: perturbing backbone coordinates yields chemically implausible geometry, so the "augmented" example is not a protein any more. Structure encoders are therefore mostly trained on task-specific labels, which are scarce. The Protein Data Bank holds roughly 182,000 experimentally determined structures against Pfam's 47 million sequences, leaving the modality that most directly determines function with both less data and the weaker pretraining recipe.

CCPL takes the pairing itself as the supervision. Every PDB entry ships a structure together with the sequence that folds into it, and a language model has already learned a great deal about that sequence. Treat the structure as a query and its sequence as the document to retrieve, train a structure encoder to rank the true partner above the other members of its batch, and the language model's knowledge transfers into the structure encoder without a single functional annotation. The sequence side is a frozen ESM-2 teacher; the artifact CCPL produces is the structure encoder trained against it.

Jiangbin Zheng and Stan Z. Li developed CCPL in the AI Lab at Westlake University, with Zhejiang University. It first appeared in March 2023 as "Lightweight Contrastive Protein Structure-Sequence Transformation" and was retitled for its publication at ICPR 2024.

#Key Features

• The sequence model teaches and stays fixed: the ESM-2 base teacher runs with frozen parameters throughout pretraining, so gradients reach only the geometric encoder. What comes out is a new structural checkpoint, not an adapted language model.
• Retrieval instead of reconstruction: a batch of B structure–sequence pairs is expanded into B² combinations, and the structure-to-sequence and sequence-to-structure cross-entropies are averaged into one symmetric alignment loss. No coordinates are corrupted, so every geometry the encoder sees is a real one.
• Two alignment granularities: one variant matches features residue by residue, the other pools them to a whole-protein vector, and the two are pretrained separately. Residue-level costs more compute and edges ahead downstream; protein-level aligns the modalities more cleanly.
• Contact maps as a structural anchor: attention maps from the encoder's self-attention blocks are symmetrized, average-product-corrected and regressed against contact maps built on virtual Cβ atoms. This ties the representation to geometry and doubles as a fine-tuning-free readout of encoder quality.

#Technical Details

The structure branch is a geometric vector perceptron (GVP) graph network — four layers over a k-nearest-neighbour graph with k = 30, a scalar hidden width of 1024 and a vector width of 256 — followed by four self-attention layers with 8 heads at width 512. It reads backbone N, Cα and C coordinates only, and the GVP formulation keeps its outputs equivariant to rotation and translation of the input frame. Pretraining used PDB sequence–structure pairs augmented with AlphaFold DB predictions, excluding anything appearing in an evaluation set, on two A100 80GB GPUs.

Frozen, the encoder reaches top-L long-range contact precision of 87.7% on trRosetta and 74.9% on CASP14. Fine-tuned as the backbone of an inverse folding model with a non-autoregressive MLP decoder, the residue-level variant records 4.48 perplexity and 50.8% sequence recovery on CATH and 60.3% recovery on Ts500, against 4.61 and 45.96% for ProteinMPNN; the same decoder trained from scratch without CCPL pretraining recovers only 39.6% on CATH. Function prediction gives Fmax of 0.459, 0.663 and 0.491 across the three Gene Ontology branches and 0.828 for EC numbers, with 97% accuracy separating enzymes from non-enzymes, and zero-shot mutation-effect ranking on ProteinGym yields an average Spearman ρ of 0.430.

#Applications

CCPL suits any task needing a structural embedding but holding too few labels to train one. Its intended downstream uses are inverse folding, where it replaces a randomly initialized backbone encoder, and function annotation — GO term, EC number, fold class and enzyme recognition — where the fine-tuning sets are small. Because contact-map prediction and sequence–structure retrieval run on the frozen checkpoint, it also serves as a structure-side representation for similarity work with no task-specific training, and the ProteinGym results extend that to fitness ranking from a backbone structure rather than from sequence.

#Impact

The pretraining scheme carried into the same group's later work: MMDesign adopts CCPL's contrastive-alignment initialization for its GVPConv structural module, taking the parameters themselves from the off-the-shelf ESM-IF structure model. The broader argument — that structural encoders can inherit from sequence models instead of competing with them — has since become a common pattern in cross-modal protein work. Reuse beyond that lineage is limited by release: no code, weights or license have been published, the authors' own publication listing offers the paper alone, and no repository exists under the group's GitHub organization. The paper specifies architecture and objectives fully, so the approach can be reimplemented, but the trained parameters stay out of reach.