ProtCLIP

Aligning protein sequences with natural-language descriptions of what those proteins do is an appealing idea: it promises a single embedding space in which a sequence can be matched to a Gene Ontology term, a disease, or a drug without training a separate classifier for each. Earlier protein-text models such as ProtST and ProteinDT established the recipe, but they inherited CLIP's coarse, whole-sequence contrastive objective and drew on the small pool of manually curated protein-text pairs. ProtCLIP, developed by Zhejiang University with Alibaba Cloud, the University of Science and Technology of China, and HKUST (Guangzhou), attacks both limitations at once.

The data problem is addressed with ProtAnno, a protein-biotext corpus assembled from SwissProt and TrEMBL. Its dense version contains 251.5 million pairs, most of them machine-annotated and therefore noisy; the sparse version holds 0.5 million manually reviewed, higher-quality pairs. Rather than discarding the noisy majority, the authors sample it with a property-driven strategy that weights each entry by annotation confidence and by how many property domains it covers, trading quality against quantity explicitly instead of by fiat.

The modeling problem is addressed by abandoning the assumption that a protein's function is spread evenly across its sequence. Function is concentrated in specific residues — active sites, binding pockets, motifs that may be contiguous or scattered through the chain. ProtCLIP adds two segment-wise objectives that target exactly those regions, alongside the usual global contrastive loss.

#Key Features

• Property-driven sampling: Selection probability is a function of annotation confidence, property coverage, and cluster size, which lets 251.5 million machine-annotated pairs contribute useful signal rather than noise.
• Static segment reconstruction: Short contiguous spans covering 15% of the sequence are masked and recovered through a cross-attention module that reads the paired functional description, forcing text to carry residue-level information.
• Dynamic segment alignment: A prototype memory bank approximates property descriptions, and residues are scored and thresholded against each prototype to form variable-length, possibly non-contiguous functional segments that are then aligned to their own property text.
• Zero-shot cross-modal transformation: Because both modalities land in one aligned space, a sequence can be matched directly against drug, disease, or GO-term candidates with no per-task fitting.

#Technical Details

The protein encoder initializes from ESM-2 at 650M parameters and the biotext encoder from PubMedBERT. Training runs on 64 Tesla V100 GPUs for roughly 10,000 GPU hours, optimizing a weighted sum of four terms — global contrastive, static segment reconstruction, dynamic segment alignment, and a masked language modeling term retained to preserve unimodal protein knowledge. The segment reconstruction and token-level MLM losses interfere strongly, and the authors report that fixed weights of 0.7 and 0.3 are needed for the loss curve to converge at all.

Evaluation spans 22 benchmarks in five families. On protein classification, ProtCLIP reports 95.08% accuracy on DeepLoc binary localization and 85.34% on the 10-category subcellular task under full tuning, with AUPR gains the authors quantify as 59.9% on GO-CC and 39.7% on GO-BP over prior state of the art. Cross-modal transformation improves by 75% on average in mean reciprocal rank across five benchmarks, reaching 0.409 MRR on Prot2Drug against 0.172 for the BioBridge baseline. On semantic similarity inference it reports Spearman correlations of 0.92, 0.88, and 0.76 for molecular function, biological process, and cellular component, against 0.33, 0.42, and 0.23 for a vanilla ESM2-3B encoder.

#Applications

The aligned space is most useful where a task has no labeled training set at all — screening an uncharacterized sequence against a vocabulary of candidate functions, ranking plausible drug associations for a target, or ordering proteins by functional similarity when the ontology annotation is missing. For groups already using protein language model embeddings as features, the released checkpoint is a drop-in alternative carrying functional annotation signal that sequence-only pretraining does not.

#Impact

ProtCLIP's contribution is the demonstration that noisy, machine-annotated protein annotation is worth pretraining on when it is sampled deliberately, and that modeling functional segments rather than whole sequences is what makes protein-text alignment competitive on fine-grained tasks. One caveat bears on how the results should be read: the reported gains come from a benchmark suite the authors assembled themselves, so they are best treated as relative improvements over the baselines they ran rather than as independently reproduced numbers.

The release is unusually complete for a protein-text model of this scale. The code is on GitHub under Apache-2.0, covering pretraining, the downstream task harness, the metrics, and the property-driven sampling filter that builds ProtAnno. A Zenodo deposit under CC BY 4.0 carries the 1.5 GB checkpoint alongside both pretraining corpora — ProtAnno-S at 537 MB and the filtered ProtAnno-D at 2.25 GB — so the model and its training data are both directly reusable. The 22 benchmark splits are not redistributed; the repository points at BioBridge's and ProtST's preparation scripts instead.