Pro-PRIME

A sequence-only protein language model learns what is typical, not what is sturdy. Two homologous enzymes — one from a soil bacterium, one from a hot-spring archaeon — are both perfectly natural sequences, so likelihood alone gives no reason to prefer the substitutions that raise a melting temperature. Pro-PRIME closes that gap by attaching a physical label to every training sequence: the optimal growth temperature (OGT) of the organism the protein came from. Predicting that number is a second pretraining objective sharing the encoder with masked language modelling, so the representation carries temperature signal alongside grammar and semantics.

A third objective ties the two heads together. For a batch of randomly generated single-site mutants, the masked-LM head scores each substitution and the OGT head predicts each mutant's temperature; the training loss maximises the Pearson correlation between the two lists. That is what the engineering use case needs — a mutation's absolute log-odds matters less than whether it ranks above the alternatives.

The model comes from Liang Hong and Pan Tan's group at Shanghai Jiao Tong University with collaborators across academia and industry, published in Science Advances in 2024 after a 2023 preprint. It ships as the Venus Prime release and is variously called PRIME, ProPrime and Prime_690M across its paper, repository and checkpoints. Within the same lab's Venus family — ProSST, ProtSSN, VenusPLM — it is the member specialising in thermal properties.

#Key Features

• Temperature as a pretraining signal: 96 million sequences carry the OGT of their host bacterial strain, so thermal tolerance is learned during pretraining rather than bolted on by a downstream regressor.
• Correlation objective: An explicit loss aligns masked-LM mutant scores with the temperatures the OGT head predicts for those same mutants, coupling two heads that would otherwise train independently.
• Zero-shot single-site scoring: The log-odds ratio between mutant and wild-type residue ranks a complete saturation landscape from one wild-type sequence, with no mutagenesis data for the target.
• Multi-site stacking: Assayed single-site results retrain the regression module, which then ranks combinations — a loop that converged on useful deep mutants in two or three rounds of fewer than 100 total variants.

#Technical Details

Pro-PRIME is a 669,208,231-parameter encoder: 33 transformer layers, hidden size 1280, 20 attention heads, a 5120-dimensional feed-forward block, rotary position embeddings and a 33-token vocabulary, matching the ESM-2 650M layout it is initialised from. The OGT head is an attention-pooling layer followed by two MLP blocks with a residual connection, emitting a temperature between 0 and 100 °C. Pretraining ran for 200,000 update steps on eight A100 80GB GPUs. The corpus, ProteomeAtlas, began as 4.7 billion sequences from UniProt and metagenomic projects, filtered to full-length entries, reduced at 50% identity with MMseqs2 and annotated with host OGT to yield the 96 million training pairs. Held-out OGT root-mean-square error was 3.5 °C and masked-LM perplexity 3.52.

Across 283 assays — 217 ProteinGym deep mutational scans plus 66 ΔTm assays drawn from MPTherm, FireProtDB and ProThermDB — the checkpoint reaches Spearman 0.486 on ProteinGym against 0.457 for SaProt, and 0.437 on ΔTm against 0.412 for Stability Oracle and 0.330 for ESM-2. Ablating the OGT head drops ΔTm to 0.362, evidence that the temperature channel rather than scale is doing the work. Fine-tuned on labels, the same encoder predicts melting temperature and optimal catalytic temperature for natural proteins from sequence alone, at Tm RMSE 4.8 °C against 5.589 °C for ESM-2 650M, and it leads ESM-1v, ESM-2 and CARP on all 12 FLIP tasks.

#Applications

The intended user is a protein engineer with a target and no screening data. Scoring the saturation landscape produces a ranked shortlist to order and assay, and the authors ran exactly that on five proteins, first fine-tuning on 30,000 homologous sequences per target with no experimental labels. Of the 30–45 top-ranked surface mutations tested per protein, roughly 30% held or raised LbCas12a melting temperature, 58% raised T7 RNA polymerase's and 32% creatinase's, 44% sped up 2'-fluoroarabino nucleic acid synthesis by the polymerase Tgo-D4K, and 38% hardened an anti-growth-hormone VHH against alkaline treatment. The stacking loop then produced a 12-site T7 RNA polymerase and an 8-site LbCas12a 6.5 °C above wild type.

#Impact

Pro-PRIME shows that a cheap, noisy environmental annotation beats adding structural input for stability prediction: the structure-aware baselines it outpaces need coordinates it does without. Its checkpoints are served through the same lab's VenusFactory tooling as a thermostability-specific alternative to general protein language models. Licensing is muddled rather than uniformly restrictive: the repository's current LICENSE file is CC BY-NC-ND 4.0, barring commercial use and redistribution of adaptations, and the HuggingFace checkpoints state no license at all — but the v1.0.0 snapshot the paper's availability statement points to was MIT (Meta's ESM license file, copyright Meta Platforms) before that swap, and the Zenodo deposit archiving it still ships the MIT file while itself declaring CC BY 4.0. That deposit also carries the evaluation material — the ProteinGym and ΔTm benchmark inputs alongside Pro-PRIME's own per-mutant predictions for all 217 ProteinGym assays. The ProteomeAtlas corpus and its OGT annotations, by contrast, have not been released, so the pretraining run cannot be reproduced. A 2025 erratum corrected mislabelled samples in one supplementary data sheet without altering the reported results.