Recursive masked protein language model whose weight-tied core reruns at test time, trading parameter count for adjustable depth.
No providers recorded yet. Browse all providers
A standard protein language model fixes its compute budget when it is built: every sequence, from a short signal peptide to a multi-domain enzyme, passes through the same number of layers, and the only way to buy more representational depth is to add parameters. recAMPLIFY separates those two quantities. A small non-recurrent prelude embeds the sequence into a residue-aligned latent state, a weight-tied Transformer core refines that state over a chosen number of cycles, and a non-recurrent coda maps the result back to masked-residue predictions. Because the core can be reapplied any number of times, effective depth becomes a setting chosen at inference rather than a property baked into the weights.
Masked encoders have no generation loop and cannot reason out loud in amino acids, so recAMPLIFY spends extra compute refining continuous hidden states instead of emitting extra tokens. The paper also introduces Gated Mixture Recurrence (GMR), which decodes each protein from a probability-weighted mixture of the states visited during recursion and uses a learned residual gate to decide how much of each cycle's update to accept, making depth adaptive per sequence.
recAMPLIFY was developed at Mila by Sarath Chandar's group with co-authors at Polytechnique Montréal, Université de Montréal, MIT's Center for Brains, Minds and Machines, and Amgen, and posted to arXiv in October 2026. It is a new architecture pretrained from scratch, not a checkpoint of AMPLIFY, but it reuses AMPLIFY's released data and training recipe so that the comparison between the two isolates the effect of recursion.
The full-scale model matches AMPLIFY-350M's hidden dimension of 960 and 15 attention heads, with a (p, r, c) = (2, 10, 2) layer split and four recurrent cycles, giving 155M unique parameters and an effective depth of 44 layers. It was pretrained for 1M steps on the same 2T tokens as AMPLIFY-350M, drawn from UniRef100, SCOP and the Observed Antibody Space. At that scale, against AMPLIFY-350M (354M parameters) retrained in the same codebase, recAMPLIFY raises CASP16 contact precision from 0.146 to 0.179, fold accuracy from 0.568 to 0.696, stability Spearman from 0.585 to 0.703 and localization accuracy from 0.716 to 0.741, ties on secondary structure (0.742), and trails slightly on ProteinGym DMS (0.348 vs 0.360) and zero-shot KL calibration.
GMR was studied with 133M-parameter models trained for 100K steps (about 210B tokens) with six cycles. All four GMR variants cut expected depth from 52 to 33–37 layer applications and improve CASP contact recovery over fixed-depth recursion, at a small cost in zero-shot DMS and KL calibration. Mechanistic analysis shows that the residual gate, not the halt head alone, drives hidden states to an empirical fixed point beyond the trained depth. The released checkpoints cover the 1M-step model, the 100K-step fixed- and variable-depth models, the four GMR variants and the retrained AMPLIFY baselines.
recAMPLIFY serves the same uses as other masked protein language models: zero-shot ranking of mutations for protein engineering and variant interpretation, unsupervised residue contact recovery, and frozen embeddings for fold classification, subcellular localization, thermostability and secondary-structure prediction. It suits groups under a memory budget: the model can run deeper when accuracy matters and shallower when throughput does, and GMR makes that trade-off per sequence.
recAMPLIFY makes test-time compute a third scaling axis for protein language models, alongside parameters and data, and its controlled comparison on identical tokens shows that a compact recursive encoder can match or exceed a model more than twice its size on most supervised probes. That optimal depth varies across ProteinGym assays, and that halt depth carries biological signal, makes adaptive computation worth studying in its own right. Code is released under an MIT license on a branch of the AMPLIFY repository, and the weights are on Hugging Face under MIT. Evaluation is entirely computational, the gains do not extend to zero-shot DMS, and the work is a preprint that has not yet been peer reviewed.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.