Tranception

BRCA1 is 1,863 amino acids long, and no alignment of the whole protein is deep or clean enough to fit a model on; the workaround is to align its RING and BRCT domains separately, which buys good predictions inside those two domains and says nothing about the rest. The trade-off is structural: a generative model built on a per-family multiple sequence alignment must be re-fit for every new protein, scores only mutations that fit its alignment's coordinate system, and degrades as that alignment thins. Around half of human proteins contain disordered stretches of at least 40 residues that resist alignment altogether.

Tranception answers with one pretrained checkpoint and no per-protein fitting step: an autoregressive transformer trained on non-aligned sequences, where the likelihood of any amino acid sequence factorizes over positions and a mutant's fitness is the log-likelihood ratio against the wild type. Because that quantity is defined for a whole sequence rather than for a masked position, insertions and deletions are scoreable out of the box. A second inference mode adds evolutionary signal without touching the weights: an alignment retrieved at inference supplies reweighted amino-acid pseudocounts per position, mixed with the model's own per-position log-probabilities. That retrieval term can be down-weighted or ignored position by position — which is what keeps indels tractable, since inserted positions have no column in the retrieved alignment.

It was built by the OATML group at the University of Oxford and the Marks lab at Harvard Medical School, with a co-author at Cohere, and presented at ICML 2022 — the same paper that introduced the ProteinGym deep mutational scanning benchmark.

#Key Features

• Zero-shot fitness scoring: A released checkpoint scores mutants of any protein directly — no assay data, no fine-tuning, no family-specific model to fit first.
• k-mer specialization in attention: Heads in each layer are split into four groups; three apply depthwise convolutions of kernel size 3, 5, and 7 to the query, key, and value projections, so heads specialize on contiguous subsequences at different scales while left-padding preserves autoregressiveness.
• Grouped ALiBi position encoding: Learned position embeddings give way to a distance-proportional attention bias applied independently per head group, cutting parameters and improving fitness prediction over the learned-embedding variant.
• Retrieval without retraining: Homolog statistics enter as a weighted term in the scoring log-likelihood, so an alignment is retrieved once per family and amortized over every mutant.
• Indels and multiple mutants: Full-sequence likelihoods extend to insertions, deletions, and deep combinatorial mutants, which alignment-bound predictors either cannot score or rank poorly.

#Technical Details

Three checkpoints were released: Small (85M parameters, 12 layers, 768-dimensional embeddings), Medium (300M, 24 layers, 1,024), and Large (700M, 36 layers, 1,280, 20 heads). All use squared ReLU activations and a 1,024-token context, wide enough to cover 98% of UniRef100 sequences; longer proteins are scored through the widest window covering the mutated positions. Training used UniRef100 — roughly 250M sequences after preprocessing, and empirically better for this task than the clustered UniRef90 and UniRef50 — with AdamW and random sequence mirroring, so scores can be averaged over forward and reverse traversals. The Large model took about two weeks on 64 A100 GPUs.

On the ProteinGym substitution benchmark (87 assays, ~1.5M missense variants), Tranception Large with retrieval reached an average Spearman correlation of 0.451 and AUC of 0.754, against 0.448 for EVE, 0.422 for the MSA Transformer, and 0.371 for ESM-1v; without retrieval it scored 0.406. The gap widens where alignments are thin: 0.453 versus 0.408 for EVE on low-depth proteins, and 0.499 on variants five or more substitutions from wild type, the highest of any model tested. On the indel benchmark (7 assays, ~300k mutants) it reached 0.463 Spearman.

#Applications

The natural uses are ranking variants for clinical interpretation across the whole proteome rather than the few thousand genes with curated family models, forecasting immune-escape mutations in viral proteins, and prioritizing candidates in protein engineering — the authors ship a Gradio app and a Colab notebook that run iterative in-silico directed evolution on top of the model. Needing no per-target training data also makes it a convenient frozen scoring function inside larger pipelines.

#Impact

ProteinGym, introduced in the same paper, has since become the reference benchmark for zero-shot variant effect prediction, and Tranception is a standing baseline in it. The architecture also seeded follow-on work from the same groups: TranceptEVE blends its autoregressive scores with family-specific models according to alignment depth, and Protriever replaces alignment-based retrieval with a learned, end-to-end differentiable homology search. Code and all three checkpoints are MIT-licensed. The limits are those of a 2022-era model: 700M parameters is modest by current standards, the 1,024-token context forces windowed scoring on long proteins, and the retrieval accuracy gain still requires an alignment to exist, even if only its aggregate statistics are used.