Loading…
Pfam-pretrained protein sequence models — transformer, LSTM and ResNet — released with five tasks benchmarking transfer from frozen embeddings.
No providers recorded yet. Browse all providers
In 2019, a paper reporting better secondary structure prediction from a protein language model could not be compared against one reporting better remote homology detection. Each group built its own splits, so an apparent gain could as easily reflect a leakier partition — homologs sitting on both sides of the split — as a better representation. TAPE (Tasks Assessing Protein Embeddings) removed that ambiguity by fixing everything except the model: one pretraining corpus, five downstream tasks with published identity-filtered splits, and one supervised head per task.
TAPE was built by Roshan Rao, Nicholas Bhattacharya, Neil Thomas and colleagues at UC Berkeley, with co-authors at covariant.ai, Google and the Chan Zuckerberg Biohub, and published at NeurIPS 2019. Alongside the benchmark the authors pretrained and released their own sequence models — a transformer, a bidirectional LSTM and a dilated residual network — each trained by masked- or next-token prediction on the same Pfam corpus. Those checkpoints, not the leaderboard, are what this entry catalogs: tape-embed maps an arbitrary FASTA to fixed-length vectors with no training step.
Two names in the release collide with other work. TAPE's transformer is exposed as ProteinBertModel and loaded as bert-base, but it is unrelated to ProteinBERT (Brandes et al., Hebrew University, 2022), which pretrains jointly on sequences and GO annotations. The checkpoint distributed as babbler-1900 is a PyTorch re-implementation of the Church Lab's UniRep, not a new model, as is the xaa–xae trRosetta ensemble.
tape-embed batches across available GPUs and writes per-sequence embeddings to an .npz file, so the checkpoints slot into ordinary regression pipelines.Pretraining uses Pfam's 31 million protein domains. About 1% — a set of complete, evolutionarily related families — is held out to measure out-of-distribution generalization, and the rest is split 95/5 at random. The transformer is 12 layers with hidden size 512 and 8 attention heads, giving 38M parameters; the LSTM is two three-layer 1024-unit LSTMs whose forward and reverse outputs are concatenated, in the style of ELMo; the ResNet is 35 residual blocks of two convolutional layers each, with 256 filters, kernel size 9 and dilation rate 2. The transformer and ResNet train with masked-token prediction, the LSTM with next-token prediction, all on four NVIDIA V100 GPUs for one week.
Self-supervised pretraining helps on nearly every task. The pretrained transformer reaches 0.73 secondary structure accuracy, 0.36 contact precision, 0.21 homology accuracy, 0.68 fluorescence Spearman ρ and 0.73 for stability; the pretrained LSTM reaches 0.75, 0.39, 0.26, 0.67 and 0.69. The negative result matters as much: alignment-based features beat every learned representation on the structural tasks, at 0.80 secondary structure accuracy and 0.64 contact precision, while learned features win decisively on remote homology (0.26 against 0.09). Task-specific methods of the era led further still: NetSurfP-2.0 at 85% CB513 accuracy, DeepSF at 41% homology accuracy. The original TensorFlow release publishes weights for all pretrained models; the maintained PyTorch rewrite ships bert-base, a 12-layer, 768-hidden, 12-head transformer retrained for that codebase, plus the UniRep and trRosetta re-implementations.
The checkpoints are used the way any frozen protein featurizer is: embed a mutant library or a homolog family once, then fit a small supervised model on top for thermostability, expression or fluorescence — far cheaper than fine-tuning end to end when only a few hundred measurements exist. The benchmark half serves model developers, who report TAPE numbers to place a new protein language model against a common reference, and the released splits spare them rebuilding identity-filtered partitions. Downstream packages also load the transformer as a pretrained backbone.
TAPE established that protein representation learning needed a shared evaluation axis, and its tasks and curated splits became a common reference point in the literature that followed. The finding that self-supervised features still trailed alignment-based features on structure prediction set the target that later, much larger models such as ESM-2 and ProtTrans were built to clear. The checkpoints themselves are orders of magnitude smaller than those successors and are no longer competitive. The authors state that the PyTorch rewrite is not meant to reproduce the paper's numbers, so those results must come from the deprecated TensorFlow repository.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.