protT5MLM

protT5MLM is a protein language model architecture released, together with a family of influenza-specific checkpoints, by virologists at the University of Tokyo's Institute of Medical Science. The starting problem is that the ProtT5-XL encoder from ProtTrans — unlike ESM-2, which ships one — has no masked language modeling head, so it cannot emit per-position amino acid probabilities. The authors attached a facsimile MLM head and trained it for two epochs on a random UniRef50 subset with the encoder frozen, giving the head sensible weights without disturbing the pretrained representation. That adapted model is protT5MLM.

With per-position probabilities available from both architectures, the authors define pLM entropy: the Shannon entropy of a model's predicted amino acid distribution at a site, conditioned on the rest of the input sequence. Unlike alignment entropy it is defined for a single sequence and is context-specific, so it can be tracked along an evolutionary trajectory rather than summarized once per alignment column.

The released artifact is a set of evotuned checkpoints. Both ESM-2 (650M) and protT5MLM were continued-pretrained by masked language modeling on unique influenza A virus hemagglutinin sequences from the NCBI Influenza Virus Database, split into serotype-specific training sets for H1, H3, H5, and H7, a combined HA-all set, and an HA-80 set restricted to the earliest 80% of collection dates (before February 2015) for forward-in-time testing. This work is a preprint and has not been peer reviewed.

#Key Features

• MLM head for the ProtT5 encoder: The added and pre-trained masked language modeling head lets the ProtT5-XL encoder produce per-site amino acid probabilities, putting it on equal footing with ESM-2 for probability-based analyses.
• pLM entropy: A per-site variability score computed from a single sequence, no alignment required, capturing how model uncertainty at a position shifts with sequence context.
• Serotype-specific evotuning: Twelve checkpoints — H1, H3, H5, H7, HA-all, and HA-80 for each base architecture — match the model to the hemagglutinin lineage under study.
• Temporal holdout by design: The HA-80 checkpoints exclude everything collected after February 2015, making their agreement with later diversity an out-of-sample test.
• Substitution-site signal: On ancestrally reconstructed internal-node sequences, evotuned models assign higher pLM entropy to sites that later changed on the descendant branch than to random unchanged sites — a separation the base models lack.

#Technical Details

The two base models are the 33-layer, 650-million-parameter ESM-2 and the 24-layer, 1.2-billion-parameter ProtT5-XL encoder, chosen as the smallest variants retaining performance comparable to their largest siblings. Evotuning is unsupervised masked language modeling on each hemagglutinin training set, and evaluation correlates per-site pLM entropy, averaged over a serotype's test sequences, against the alignment entropy of the corresponding multiple sequence alignment.

The HA-all checkpoints of both architectures reach the most consistent agreement, with Spearman coefficients between 0.83 and 0.89 across the four test serotypes. Base ESM-2 correlates moderately and unevenly — 0.74 on H1 and 0.67 on H5 versus 0.43 on H3 and 0.45 on H7 — tracking how well each serotype is represented in the UniRef50 cluster it was trained on. Base protT5MLM shows essentially no correlation (as low as -0.06), but once the encoder and the added head are evotuned it becomes directly comparable to the evotuned ESM-2 models. Single-serotype checkpoints match or exceed HA-all on their own serotype but degrade sharply across serotypes, falling to 0.25 for the ESM-2 H5 model on H7 sequences; the drop tracks pairwise amino acid similarity, with the distant H3 and H7 serotypes (~40% similarity) transferring worst. The HA-80 checkpoints perform nearly identically to HA-all on post-2015 sequences they never saw.

#Applications

The checkpoints apply directly to influenza A hemagglutinin surveillance: scoring which residues of a circulating strain are most likely to change next, tracking how site variability shifts along a phylogeny, and comparing conservation between lineages without building an alignment each time. The paper demonstrates the workflow on the human H3N2 backbone, where pLM entropy at sites 157 and 162 rises in the years immediately preceding the observed K-to-I and N-to-S substitutions. The evotuning recipe and the entropy metric port to any protein group with enough sequence data, and the added MLM head makes ProtT5-family encoders usable for probability-based evolutionary inference generally.

#Impact

The practical contribution is a demonstration that targeted continued pretraining closes a large gap: base protein language models trained on UniRef represent hemagglutinin diversity poorly and unevenly, and evotuning on family sequences brings both architectures to high agreement with alignment-derived conservation. That transfer degrades with sequence divergence is a useful caution for anyone reusing family-specific checkpoints. Scope is the main limitation: these are influenza hemagglutinin models, not general protein models, and the entropy signal for future substitutions is a distributional shift rather than a calibrated per-site predictor. The Zenodo archive is CC-BY-4.0, but the GitHub repository carries no license file, and hemagglutinin sequences sourced from GISAID are excluded from it under that database's terms.