Ankh

Ankh is an optimized protein language model (PLM) developed by Ahmed Elnaggar, Burkhard Rost, and colleagues at the Technical University of Munich. First posted as a preprint in January 2023 and published in the Computational and Structural Biotechnology Journal in 2024, it targets a central assumption in the field: that performance requires ever-larger models. Systematically investigating masking strategies, architecture design, and training data composition across more than twenty configurations, the team produced a model that reaches state-of-the-art results on protein prediction tasks with fewer than 10% of the parameters needed to train comparable models such as ESM-2 (15B).

Ankh Large (~1.15 billion parameters) improves the average PLM benchmark by 4.8% over prior state-of-the-art models, and Ankh Base (~450 million parameters) yields a 3.4% improvement using only 3% of the training parameters of leading alternatives. That efficiency translates into accessibility: Ankh Large runs on a single A100 40 GB GPU, versus the four A100 80 GB GPUs ESM-2 requires at equivalent sequence lengths.

An Ankh2 series (Ankh2 Large, ~2 billion parameters) extended the original with more training epochs and architectural refinements. Ankh3, described in a May 2025 preprint by Hazem Alsamkary, Mohamed Elshaffei, Mohamed Elkerdawy, and Ahmed Elnaggar, changes the pre-training recipe itself: rather than denoising alone, it is jointly optimized on masked language modeling at multiple masking probabilities and on autoregressive sequence completion, still taking only sequence as input. All variants, including the Ankh3 Large and Ankh3 XL checkpoints, are on HuggingFace under the ElnaggarLab organization, licensed CC BY-NC-SA 4.0 (non-commercial).

#Key Features

• Efficiency-first design: Ankh Large matches state-of-the-art benchmarks with under 10% of the pre-training and under 7% of the inference parameters of ESM-2 (15B), enabling single-GPU deployment.
• T5-based encoder-decoder architecture: Unlike encoder-only PLMs, Ankh uses a T5-style encoder-decoder transformer with 1-gram span de-masking at 20% masking probability, which systematic ablations found superior to other masking strategies.
• Significant inference speedup: Feature extraction runs 2.2x to 11.7x faster than ESM-2 depending on sequence length, cutting the cost of embedding large screening libraries.
• Multiple model scales: Four public variants (Ankh Base, Ankh Large, Ankh3 Large, Ankh3 XL) let users match capacity to available compute and task requirements.
• Multi-task pre-training in Ankh3: The 2025 generation combines masked-language denoising at 15%, 20%, and 30% masking with autoregressive sequence completion, yielding stronger representations than denoising alone.

#Technical Details

Ankh Large has an embedding dimension of 1536, 48 encoder layers, 24 decoder layers, 16 attention heads, and a feed-forward dimension of 3840; Ankh Base uses 768, 12 heads, and 3072. Both use Gated-GELU activations and relative positional encodings (offset 128, dimension 64). Pre-training used UniRef50 (~45.6 million sequences) on Google TPU-v4 pods; lower-redundancy databases produced better downstream representations than UniRef90 or UniRef100.

Ankh Large achieves 83.6% Q3 on CASP12 secondary structure prediction and 61.0% on SCOPe remote homology fold classification (1,194 classes), improves on TAPE solubility and fluorescence, and excels at embedding-based contact prediction relative to earlier attention-map approaches. Ankh2 Large (~2 billion parameters) continued from the Ankh Large checkpoint for 45 epochs with Adafactor and linear warmup, substituting SiLU for GELU. Ankh3 scales further: Ankh3 Large totals 1.88 billion parameters (1.15B encoder, 727M decoder) and Ankh3 XL 5.73 billion (3.48B encoder, 2.25B decoder), both pre-trained on 59.1 million distinct UniRef50 sequences with JAX and T5X on 64 TPUv4 chips. Using Ankh3 XL encoder embeddings, the preprint reports 84.4% Q3 on CASP12 and 82.2% on CASP14 (versus 77.5% for Ankh Large and ESM-2 15B), Spearman correlations of 0.642 on fluorescence and 0.896 on GB1 fitness, and 61.0% L/1 precision on ProteinNet contact prediction against 48.9% and 43.2% respectively.

#Applications

Ankh is a general-purpose protein representation model for fine-tuning or fixed-feature extraction. Researchers use it for structure prediction, function annotation, fitness landscape modeling, and variant effect prediction, and its efficient inference suits proteomics workflows needing embeddings for hundreds of thousands of sequences. Parameter-efficient fine-tuning via LoRA has been demonstrated for solubility and fluorescence tasks. The family also suits protein engineering, where capturing conservation and mutation trends supports generating diverse yet functionally coherent variants; Ankh3's completion objective additionally makes it usable as a generative continuation model rather than an encoder alone.

#Impact

Ankh is a substantial contribution to the argument that protein-specific inductive biases, rather than raw scale, drive PLM performance. By matching or beating models ten times its size, it shifted community attention toward efficient training strategies and knowledge-guided architecture choices, and it runs on commodity GPU hardware, making it practical for groups without large-scale compute. The 2025 release of Ankh3 extends that argument from architecture to training objectives, showing that combining denoising with sequence completion yields stronger representations than denoising alone. Two limitations temper this. Ankh, like other sequence-only PLMs, sees no structural information during pre-training, so structure-aware models remain complementary. And the CC BY-NC-SA 4.0 license on every released checkpoint restricts commercial deployment, narrowing industrial adoption relative to permissively licensed alternatives.