ProteinAligner

A protein language model reads an amino acid sequence and nothing else. For a Swiss-Prot entry like human insulin, that discards two other experimentally determined descriptions of the very same molecule: the deposited PDB structure, and the reviewed prose stating that it lowers blood glucose and increases cell permeability to monosaccharides. Both were curated by domain experts, and both are invisible to sequence-only pretraining.

ProteinAligner, from UC San Diego with collaborators at Scripps Research, Carnegie Mellon and MBZUAI, closes that gap by pretraining three encoders jointly and tying them together with contrastive alignment. Its design decision is which modality gets to be the reference point. Rather than requiring complete (sequence, structure, text) triples, ProteinAligner makes the amino acid sequence the anchor and aligns the other two to it independently — a sequence-structure loss and a sequence-text loss, summed with equal weight. Any protein that has a sequence plus at least one other modality contributes to training. That matters because modality coverage is badly lopsided: in the curated corpus, 290,480 proteins carry a reviewed textual description, but only 133,726 of those also have a solved structure.

The framework sits alongside two-modality predecessors — ProtST pairs sequence with biomedical text, ESM-IF1 pairs sequence with structure — and is the three-modality generalization of both. A separately developed contemporaneous study explored the same three-modality idea; the authors note the two lines of work were conceived independently.

#Key Features

• Sequence as the alignment anchor: structure and text embeddings are each pulled toward the sequence embedding, never toward each other, so no protein needs all three modalities present.
• Reuse of pretrained modality encoders: the sequence pathway is an ESM protein language model and the structure pathway is the ESM-IF1 encoder with its inverse-folding decoder discarded; only the text encoder is trained from scratch.
• Expert-curated pretraining corpus: pairs are built by mapping PDB entries to UniProt accessions, so both the structures and the functional descriptions are experimentally validated and manually reviewed rather than predicted.
• Per-modality downstream reuse: tasks that take sequences fine-tune the aligned sequence encoder; tasks that take atomic coordinates fine-tune the aligned structure encoder, each with a small task-specific head.

#Technical Details

ProteinAligner totals 867 million parameters. The sequence encoder is a 33-layer ESM transformer with embedding dimension 1280 and 650 million parameters; the structure encoder, taken from ESM-IF1, stacks four Geometric Vector Perceptron graph-neural-network layers for geometric features under eight transformer encoder layers, at 124 million parameters; the text encoder is an eight-layer transformer with 78 million parameters. Linear projection modules map all three into a shared latent space. Pretraining minimizes two InfoNCE losses at temperature 0.07 over 20 epochs on 40 A100 GPUs with AdamW, an initial learning rate of 5e-6 and cosine-annealing decay. The corpus draws on UniProtKB/Swiss-Prot release 2023_02 and the 2 May 2023 PDB snapshot of 200,734 structures, with sequences longer than 300 residues excluded.

Across five task families the margins are consistent but generally small. On type I anti-CRISPR activity detection over 227 Acr/Cas pairs, ProteinAligner reaches AUC 0.852 and F1 0.84 against ESM at 0.732 and 0.77 and ProtST at 0.828. On VariPred missense pathogenicity it scores F1 0.71 versus 0.667 for ESM and 0.676 for ProtST, though the test split is only 100 examples. On five-class thermostability over the 1,040-protein HP-S²C5 set — the one task driven by the structure encoder — it reaches F1 0.608 and accuracy 0.577 against ESM-IF1's 0.559 and 0.542. Eight peptide bioactivity tasks and a minimum-inhibitory- concentration regression on E. coli antimicrobial peptides round out the evaluation; blood-brain-barrier penetration improves to AUC 0.824 from 0.788 (ESM) and 0.767 (ProtST).

#Applications

The evaluated tasks are all low-label function and property calls where a general-purpose representation substitutes for training data that does not exist: flagging anti-CRISPR proteins when engineering CRISPR-based tools, scoring clinical missense variants for pathogenicity, ranking enzyme variants by thermostability for industrial biocatalysis, and triaging candidate therapeutic peptides by blood-brain-barrier penetration, antiviral or antiparasitic activity, and antimicrobial potency.

#Impact

ProteinAligner is a bioRxiv preprint with a single version and no journal publication, and its results are computational throughout — no wet-lab validation accompanies the benchmark gains. The pretraining and downstream code is public under CC BY-NC-SA 4.0, and a Zenodo software archive mirrors it, but the pretrained checkpoints the paper's code-availability statement points to are not present in the repository, and two requests for them filed as GitHub issues remain unanswered. The contribution that survives regardless is architectural: anchoring on the most universally available modality is what lets a multi-modal protein model train on the ragged, incomplete coverage that real protein databases actually have.