ProtJEPA

More than 99.9% of known protein sequences carry no experimentally validated functional annotation. The information that would characterize them — solved structures, curated interaction networks, expression atlases, published literature — exists in abundance for the small, well-studied fraction of the proteome and is almost entirely absent for the rest. Methods that consume those signals directly therefore work least well exactly where annotation is most needed.

ProtJEPA, developed at MIT by Vaibhava Lakshmi Ravideshik, Jinha Kim, and Manolis Kellis and posted as a preprint in August 2026, attacks that asymmetry by moving the multimodal information into the model's weights rather than its inputs. It is a Joint-Embedding Predictive Architecture: a sequence-only student encoder is trained to predict a joint embedding assembled from ten biological modalities — sequence, structure, knowledge graph, protein interactions, literature, subcellular localization, tissue expression, GO function, anatomy, and disorder. Each modality has its own teacher encoder, and a modality-attentive aggregator fuses their outputs into the single target the student learns to reproduce. At inference the auxiliary modalities are gone; an amino acid sequence is all the model needs.

The JEPA framing — predicting in representation space instead of reconstructing inputs — has been carried into biology by models such as GeneJEPA for single-cell transcriptomics and JEPA-DNA for genomic sequence. ProtJEPA extends the idea in a different direction: its prediction target is not a masked view of the same modality, but a fused representation of nine other kinds of evidence about the same protein.

#Key Features

• Sequence-only at inference: The student encoder consumes an amino acid sequence and nothing else, so the model applies to proteins with no structure, no known interaction partners, and no literature.
• Ten-modality joint target: Structural, relational, textual, and expression evidence are combined into one prediction target rather than handled by separate task heads.
• Modality-attentive multi-teacher fusion: An aggregator weights the teacher encoders when building the joint target, instead of naively concatenating their embeddings.
• Target whitening: Joint targets built from multiple teachers are severely anisotropic — mean pairwise cosine similarity of 0.984. Whitening drops that to 0.086 and prevents representation collapse without covariance regularization.
• Dark-protein evaluation protocol: Results are reported on 1,828 held-out proteins with zero primary Pfam family overlap with the training set, so the measured transfer cannot come from family memorization.

#Technical Details

Training proceeds in two phases: the modality aggregator is pretrained first, then the sequence-only student is trained against the whitened joint targets. Ablations establish that Phase 1 aggregator pretraining and target whitening are each independently load-bearing — removing either degrades downstream performance on its own.

On the dark-protein split, ProtJEPA reaches 58.07% Hit@10 on zero-shot GO term retrieval (+2.80 percentage points, p=0.020) and 69.99% enzyme class accuracy (+9.64 pp, p<0.001), and improves subcellular localization by +11.87 pp when only 1% of labels are available for fine-tuning (p<0.001). Under deployment conditions where relational modalities are unavailable for a query protein, it outperforms naive concatenation of the remaining modalities. Cross-domain evaluations on drug-target interaction and disorder prediction — tasks outside the training modalities — confirm that the representations transfer, with a consistent ordering of single-teacher baseline below the ESMC protein language model below ProtJEPA replicated across six independent tasks.

#Applications

ProtJEPA targets the dark proteome: proteins from metagenomic surveys, newly sequenced genomes, and understudied organisms that homology transfer and structure-dependent annotation pipelines handle poorly. Its embeddings support zero-shot GO term retrieval and enzyme class assignment, and serve as a starting point for low-label fine-tuning where only a handful of characterized examples exist — the regime most functional genomics groups actually work in. Because the frozen encoder also transfers to drug-target interaction and intrinsic disorder prediction, one representation can back several downstream screens without retraining the backbone.

#Impact

The contribution most likely to outlast the specific checkpoint is methodological: ProtJEPA shows that heterogeneous, incomplete relational biology — knowledge graphs, interaction networks, expression atlases, text — can be compressed into a sequence-only encoder, and it identifies target whitening as the mechanism that makes multi-teacher joint targets trainable at all. The paper's evaluation design is also unusually strict for the subfield, since enforcing zero Pfam family overlap removes the homology leakage that inflates many protein function benchmarks. Its practical reach is limited for now: the work is a preprint that has not been peer reviewed, no code or trained weights have been released, and the preprint carries a non-commercial CC BY-NC license.