Loading…
Protein language model family from 8M to 15B parameters, used as a frozen sequence encoder whose representations encode atomic-level structure.
ESM-2 is a family of transformer protein language models built by Meta AI to test how much of a protein's three-dimensional structure can be recovered from sequence statistics alone. Six encoders spanning 8 million to 15 billion parameters were trained on a single objective — predict amino acids masked out of a sequence — and were never shown a structure. Lin et al. reported the outcome in Science in 2023: the larger the model, the sharper the picture of structure sitting in its internal representations.
The premise is that evolution records geometry into sequence: selection acts on a protein's fold, so the pattern of substitutions tolerated across millions of related sequences carries the constraints on how a chain packs. A model that fills in masked residues well must internalize part of that geometry. ESM-2 makes the relationship measurable — across the family, validation perplexity correlates -0.99 with CASP14 TM-score and -1.00 with CAMEO TM-score.
That finding is what makes ESMFold possible, a folding head trained on the 3B ESM-2 encoder that predicts all-atom structure from a single sequence. But ESM-2's day-to-day role is broader: the released encoders became the default representation layer for protein machine learning, and a large share of the models in this catalog build on a frozen ESM-2 checkpoint as their sequence tower.
ESM-2 is an encoder-only transformer trained with a masked language modeling objective, scaling from 6 layers and a 320-dimensional embedding at 8M parameters to 48 layers and 5,120 dimensions at 15B. Relative to ESM-1b it swaps learned positional embeddings for rotary position embeddings, drops dropout entirely to free capacity, and raises the effective batch to 2 million tokens, 3.2 million for the 15B model; long proteins are cropped to 1,024 tokens during training. All models ran for 500K updates, the 15B for 270K.
Structural content is measured at two resolutions. A linear projection of the attention maps recovers residue-residue contacts, and long-range precision at top-L rises from 0.16 at 8M to 0.54 at 15B, with proteins that have many evolutionary relatives saturating at smaller scales while sparsely represented ones keep improving as the model grows. At atomic resolution, an equivariant transformer fit on PDB structures projects coordinates directly out of the frozen representations; the 15B model reaches a TM-score of 0.72 on a 194-protein CAMEO test set and 0.55 on 51 CASP14 targets, gains of 14% and 17% over the 150M model. Both evaluations use a temporal cutoff, so test proteins are held out from the projection fit.
As a pretrained sequence encoder, ESM-2 supplies general-purpose representations for function annotation, variant effect prediction, protein-protein interaction prediction, remote homology detection, fitness landscape modeling, and scoring designed sequences. The usual pattern is to freeze a checkpoint and train a light head — a classifier, a regressor, or a graph network — on its per-residue or mean-pooled embeddings, which puts these tasks within reach of a few thousand labeled examples. The 650M and 3B checkpoints are the workhorses, while the 8M and 35M models are cheap enough to embed whole proteomes. Community extensions address the residue-length ceiling, among them ESM2 Long, which re-pretrains the 33-layer configuration with local windowed attention over 2,048-residue inputs.
ESM-2 shifted the argument about protein language models from whether they work to why: structural knowledge turns out to be a predictable consequence of sequence modeling fidelity at scale. Its encoders are among the most widely adopted protein representations in the literature and serve as the sequence backbone for a large share of the downstream models in this catalog. The honest caveat is that representation quality tracks the model's own fluency on a sequence, so proteins from sparsely sampled corners of sequence space are served least well. The line continued with ESM-3, which reasons jointly over sequence, structure, and function, and with ESM Cambrian, which returns to the pure encoder and targets representation quality per parameter.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.