Loading…
Protein language model that grows its own depth and width during masked-language pretraining on the human proteome, reaching 76.0% Pfam-50 accuracy.
No providers recorded yet. Browse all providers
Every mainstream protein language model fixes its shape before the first gradient step. ESM-2 and ProtBERT, from the ProtTrans suite, each commit to a depth, a width and a head count chosen by analogy with prior work, then spend tens to hundreds of millions of sequences filling that shape in. Nothing during training says whether the shape was right: if the signal reachable through masked-language modelling is lower-dimensional than the encoder assumes, much of the parameter budget sits unused, and if it is higher-dimensional, the encoder is too small.
INCRT-geo inverts the order, starting from a minimal encoder that grows itself from quantities it can already measure. A new attention head is inserted when the largest eigenvalue of a layer's residual attention operator crosses a threshold — when spectral mass concentrates on a direction the current head pool does not span. A head whose contribution falls below the counterpart threshold is pruned, and a new layer is stacked on top when the deepest layer's per-head asymmetry index stays above 0.9 for 200 consecutive steps. Every insertion is soft-identity initialised, so the augmented network reproduces the forward pass it replaced.
The study is the first application of the three-level INCRT-geo framework to biological sequences, from Giansalvo Cirrincione at the Université de Picardie Jules Verne with Elisa Ficarra and Marta Lovino at the University of Modena and Reggio Emilia, posted as a bioRxiv preprint in September 2026.
The encoder is a BERT-style masked-language model over a 24-symbol character-level vocabulary — the twenty canonical amino acids plus four special tokens — trained with the standard 15% masking recipe at sequence length 512. The corpus is the Ensembl release 115 human proteome, filtered to 10–510 residues, stripped of non-standard residue symbols and deduplicated to about twenty thousand sequences. The principal grown model, denoted v9 in the paper, ends at 22.7M parameters in four layers with head counts [207, 135, 1, 1]; the two single-head top layers act as compositional readouts over the wide feature-extracting layers beneath. On Pfam-50 family classification it reaches 76.03% linear-probe accuracy (75.04 ± 1.82 across three seeds) and 70.78% under full fine-tuning, against 71.13% for ESM-2 small (7.5M parameters) and 72.18% for ProtBERT (420M) under an identical protocol — a comparison the authors present as holding the evaluation protocol constant, not as a parameter-matched contest. The ablations cost 12.45 points of probe accuracy for 3-residue tokenisation, 6.05 for disabling depth growth and 8.56 for removing the asymmetry loss; in the last case the layer-0 asymmetry index peaks at 0.709, the depth trigger never fires, and the model collapses to a single wide layer. A variant pretrained on 283,891 sequences from eight vertebrate proteomes reaches 55.94% probe accuracy, on a run truncated at two epochs by memory limits and therefore not converged.
The result is aimed at groups whose biological question is already scoped — one organism, one tissue, one clinical cohort — and who would otherwise adopt a general-purpose pLM they cannot retrain. One carefully pretrained proteome plus on-demand capacity allocation yields a frozen encoder useful for family-level annotation at a budget of a few GPU-hours. The growth machinery is also an instrument in its own right: the asymmetry index, the effective rank of the antisymmetric head stack and pairwise head similarity give a vocabulary for reading how much capacity a corpus warrants.
The claim is deliberately narrow. Evaluation covers Pfam-50 family classification and nothing else — no contact prediction, secondary structure, localisation or function annotation — and the corpus is human-restricted, which is also the distribution the test split is drawn from. The multi-vertebrate run is reported as a negative result rather than a scaling success, with the caution that taxonomic breadth is not free when the downstream probe stays single-species. What the study establishes is that architecture can be an output of pretraining rather than an input, and that the asymmetry-loss regulariser is load-bearing: without it the growth criterion never becomes actionable. The code and the five pretraining notebooks are released under Apache 2.0; the pretrained checkpoints are announced for a Zenodo deposit upon journal acceptance and are not yet available. The work has not been peer reviewed.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.