Loading…
Suite of six protein language models, including ProtBERT and ProtT5, trained on up to 393 billion amino acids without multiple sequence alignments.
Before 2020, giving a sequence-based predictor evolutionary context meant a database search for every query protein to build a multiple sequence alignment — minutes of compute per sequence, and little to work with when a protein has no detectable relatives. ProtTrans asked whether a language model trained on raw amino acid sequences could absorb those same constraints from co-occurrence statistics alone and deliver them in one forward pass.
Ahmed Elnaggar, Michael Heinzinger, Burkhard Rost and colleagues at Rostlab (Technical University of Munich) trained six protein language models on up to 393 billion amino acids from UniRef and the Big Fantastic Database (BFD), using 5,616 GPUs on the Oak Ridge Summit supercomputer and up to 1,024 Google TPU cores. The suite spans two families: auto-regressive models (ProtTXL, ProtXLNet) and auto-encoder or encoder-decoder models (ProtBERT, ProtAlbert, ProtElectra, ProtT5). Scoring all six through a shared downstream pipeline made the architecture comparison, not any single checkpoint, the contribution.
The answer was affirmative: embeddings from ProtT5, the largest model, pushed three-state secondary structure prediction past the alignment-based state of the art on the NEW364 test set with no alignment input, the first time a protein language model had done so. It appeared in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2021, alongside Facebook AI's ESM-1b.
prot_bert on HuggingFace, is the 420-million-parameter BERT member of this Rostlab suite. ProteinBERT is an unrelated 16-million-parameter model from Brandes and colleagues that folds Gene Ontology annotation prediction into pretraining. Papers naming "a BERT protein language model" usually mean the former.The flagship, ProtT5-XL-UniRef50, is a 3-billion-parameter encoder-decoder built on T5-3B and pretrained with a BART-style masked objective at 15% masking on UniRef50 (roughly 45 million sequences), initialized from a ProtT5-XL-BFD checkpoint trained on BFD (2.1 billion sequences). Tokenization is per residue, with the rare symbols U, Z, O and B mapped to X; the final encoder layer emits 1,024-dimensional vectors, mean-pooled for per-protein tasks.
ProtBERT matches BERT-large's width — hidden size 1,024, 16 attention heads, intermediate size 4,096 — but stacks 30 layers for 420 million parameters, with a positional-encoding limit of 40,000 residues. Next-sentence prediction is dropped and each protein is treated as a single document, leaving a 15% masked-language-modeling objective trained with LAMB on a TPU Pod. Two checkpoints exist: ProtBERT on UniRef100 (prot_bert, 300k steps at length 512 then 100k at 2,048) and ProtBERT-BFD on BFD (prot_bert_bfd, 800k then 200k steps). On CASP12 they score Q3 = 75.0 and 75.8 respectively, and both reach Q10 = 74 for localization and Q2 = 89 for membrane versus water-soluble. ProtBERT is the suite's fastest at inference, averaging 0.007 seconds per protein at batch size 32 on one Quadro RTX 8000 in half precision, 16 to 28 times faster than building an MSA with MMseqs2. The other checkpoints keep the same naming: prot_albert (224M), prot_xlnet (409M), prot_electra_generator_bfd and prot_electra_discriminator_bfd (420M), and prot_t5_xl_uniref50. Every benchmark figure comes from a lightweight head trained on frozen embeddings, not end-to-end fine-tuning.
ProtTrans embeddings are a standard feature representation in protein bioinformatics, serving as input to secondary structure prediction, localization classifiers, stability and variant effect prediction, and fold recognition. SignalP 6.0 is the clearest production tool built on the suite: it takes the 30-layer UniRef100 ProtBERT encoder, replaces its final layer, extends the vocabulary with organism-group tokens, and decodes with a conditional random field to detect all five classes of signal peptide. Because the checkpoints load through the HuggingFace Transformers API, feature extraction is a single forward pass and a few lines of code, within reach of labs with no dedicated ML infrastructure.
ProtTrans is among the most cited protein language model papers. It made large-scale self-supervised pretraining on protein databases routine and showed that representations learned without structural supervision carry enough signal for competitive downstream prediction. Its checkpoints stay in service as frozen backbones: ProstT5 extends ProtT5 to translate between amino acid and 3Di structural alphabets, and protT5MLM evotunes the encoder onto influenza hemagglutinin. Limitations follow from the design: each sequence is modeled independently, with no alignment or inter-chain context, and the largest checkpoints (ProtT5-XXL, 11B parameters) demand substantial GPU memory, which is why the 420M ProtBERT remains the practical choice for high-throughput work.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.