CAMP (Contrastive Annotation Model for Proteins)

A protein language model trained by masked language modeling over amino acids learns a latent space organized largely by structure: two sequences that fold alike land near each other whether or not they catalyze anything alike. That is a poor coordinate system for what annotation work actually asks: what a protein does. CAMP's premise is that biologists have already written the answer down in machine-readable form — the EC numbers, Gene Ontology terms, InterPro families and Gene3D domains curated in UniProt — and that those ontologies can be modeled directly if each member gets a token. The Annotation Vocabulary maps every member to a unique integer, yielding a 30,000–60,000 token vocabulary a transformer reads the way it reads residues, with no natural-language filler standing in for function.

CAMP — Contrastive Annotation Model for Proteins — was built by the Gleghorn Lab at the University of Delaware and released as a preprint in July 2024. It uses that vocabulary to reshape an existing protein latent space rather than train a new one. An Annotation Transformer (AT) is first pretrained by masked language modeling on annotations alone, never seeing a sequence. AT and ESM-2 650M are then both frozen, and only a one-block ConvBERT plus linear projections on each track are trained, with a contrastive objective that is deliberately indirect: rather than pull a sequence toward its own annotation vector, the loss matches the distribution of sequence-to-sequence similarities in a minibatch to that of annotation-to-annotation similarities. Because everything large stays frozen, the premier variant cost about three hours on one A6000. The same preprint builds three further architectures on the vocabulary: AT, the dual-vocabulary ASM and the generative GSM.

#Key Features

• Ontology terms as first-class tokens: EC, GO cellular component, molecular function and biological process, InterPro and Gene3D members each get an integer, fed in sorted order so the model learns an annotation "grammar".
• Both towers frozen: Only small projection heads train, keeping the base representations intact and latent space curation cheap enough to iterate on.
• Sequence-only at inference: Only the sequence track runs once training is done, so CAMP is a drop-in encoder and downstream users need no annotations at all.
• Three released variants: CAMP_EXP, CAMP_RED and CAMP_NAT differ in training corpus, the NAT variant swapping AT for a frozen SciBERT over natural-language descriptions as a control.

#Technical Details

AT is a single BERT-like block with hidden size 384 and intermediate dimension 2,048, with rotary position embeddings and 15% masking; its vocabulary is 33,328 tokens on EXP and 38,953 on RED, both built with Pfam standing in for InterPro. The three CAMP corpora are EXP (70,395 UniProt sequences with experimentally validated annotations), RED (516,184 UniRef90 representatives) and NAT (1,435,224 UniRef50 sequences paired with natural-language descriptions). Each variant trained for one epoch; the released CAMP_EXP checkpoint holds 674 million parameters, 658 million of them the inference-time sequence track.

Across 15 standardized datasets fed to a linear probe on frozen embeddings, CAMP_EXP averaged an F1 of 0.601 on the in-distribution tasks — the only model tested above 0.6 — against 0.586 for ESM3, 0.589 for Ankh base and 0.579 for its own base ESM-2 650M. On tasks the authors label out-of-distribution, CAMP variants took the top two places on both yeast and human protein-protein interaction prediction despite never being trained for it, while falling below base ESM-2 on metal binding. Assigning EC numbers zero-shot by vector search against a SwissProt reference database, CAMP_EXP reached 0.785 average AUC over the CLEAN New, Price and Halogenase benchmarks, within 0.001 AUC of the best reported score on two of the three.

#Applications

CAMP is aimed at the annotation gap: most sequenced proteins carry no experimental characterization, and embedding transfer is the cheapest way to propose one. Because inference needs only a sequence, a group can embed an unannotated proteome once and then train small probes for enzyme class, localization or GO terms on whatever labels it has, or skip supervision and assign EC numbers by nearest-neighbor search against SwissProt. The recipe generalizes: any lab holding a curated ontology can reshape a pretrained encoder's latent space for a few dollars rather than fine-tune it.

#Impact

The limits are the ones the authors draw themselves. Gains hold only for properties the Annotation Vocabulary represents: CAMP_EXP underperforms its base model on metal binding precisely because cofactor annotations are sparse in EXP and absent from RED. The work remains a preprint that has not been peer reviewed, its evaluation is entirely in-silico, and its probe protocol measures how embeddings correlate with a task, not how the model fine-tunes for one. Its influence is visible downstream: the Annotation Vocabulary reappears as a conditioning signal in the lab's later DSM diffusion model, and the Atlas-CAMP lane of Synthyra's hosted Atlas service grows out of this line of work, on different weights.