ESM2 Long & ESM2 Quantized

The ESM-2 protein language models accept at most 1,022 amino acids — 1,024 tokens once the CLS and EOS markers are counted. That ceiling is an architectural consequence of global self-attention, whose cost grows with the square of sequence length, and it forces anyone working on large proteins to truncate them, discard them, or chop them into windows and stitch the resulting embeddings back together. Titin, dystrophin, and a long tail of multidomain enzymes and structural proteins fall outside the limit.

ESM2 Long and ESM2 Quantized are re-pretrained checkpoint families from the Institute of Computing at the University of Campinas that raise the ceiling to 2,048 amino acids. Following the LongFormer recipe, Gabriel Bianchin de Oliveira, Helio Pedrini, and Zanoni Dias replaced ESM-2's global attention with local windowed attention, extended the context representation to 2,050 positions, and pretrained the modified networks on Swiss-Prot. The quantized family repeats the exercise in 4-bit integer precision using LoRA adapters, trading a little accuracy for a much smaller memory footprint.

These are genuinely separate weights, not a wrapper around ESM-2. The work was presented at the XVII Brazilian Symposium on Bioinformatics (BSB 2024) and posted to arXiv in January 2025.

#Key Features

• Doubled context window: Each model reserves 2,050 token positions — 2,048 for amino acids plus CLS and EOS — against the 1,024 of the original architecture, so most long proteins are embedded in a single forward pass.
• Windowed local attention: Every token attends only within a 1,024-token window, moving the attention cost from O(n²) to O(nk). Even at the doubled input length, this roughly halves what global attention over 2,050 tokens would require.
• Context copying over random initialisation: The positional context representation is copied out to the new length rather than randomly initialised, a choice carried over from the LongFormer results and one reason five epochs of pretraining suffice.
• Four-bit quantized variants: The quantized family is trained in int4 with bfloat16 compute and distributed as LoRA adapters over companion base checkpoints. The 33-layer model loads in 664 MB against 3,338 MB for the full-precision long version.
• Size coverage across the ESM-2 family: Long checkpoints exist for the 6-, 12-, 30-, and 33-layer configurations; quantized checkpoints add the 36-layer configuration. Compute constraints ruled out a 36-layer long model and any 48-layer conversion.

#Technical Details

Each modified architecture keeps ESM-2's layer count, head count, and embedding width — 33 layers, 20 heads, and a width of 1,280 for the largest long model — and changes only the attention pattern and the context length. Pretraining ran for five epochs over all 569,793 proteins in the July 2023 Swiss-Prot release, chosen over TrEMBL for its manual curation, with AdamW and a learning rate of 1e-5.

Evaluation used protein function prediction on a CAFA5-derived benchmark, with roughly 74,000 training proteins and 500 Gene Ontology terms per sub-ontology. Last-layer embeddings were extracted with the backbone frozen and fed to an AutoKeras classifier selected over 50 trials, so the numbers isolate representation quality. Across the whole test set the 33-layer models reach an Fmax of 0.549 (quantized) on Biological Process, 0.751 (long) on Cellular Component, and 0.783 (quantized) on Molecular Function, against 0.540, 0.736, and 0.773 for standard ESM-2. Restricting the test set to proteins longer than 1,024 amino acids — where the modification should matter most — widens the gap: 0.556 versus 0.517 on Biological Process for the 33-layer quantized model, and 0.533 versus 0.493 for the 12-layer long model. The advantage is not uniform; standard ESM-2 still wins Biological Process at 30 layers on the full test set.

#Applications

The checkpoints are aimed at anyone whose protein set includes sequences the original ESM-2 cannot ingest whole. Function annotation, family assignment, and conservation analysis all consume ESM-2 embeddings, and all currently pay a windowing penalty on the roughly 10–12% of proteins in a typical eukaryotic set that exceed 1,024 residues. Because the models retain the ESM-2 interface and load through the standard Transformers masked-language-model API, swapping them into an existing embedding pipeline is a one-line change. The quantized variants extend that to hardware where the full-precision 33-layer model would not fit.

#Impact

This is a focused engineering contribution rather than a new modelling paradigm, and it is scoped accordingly: a single downstream task, a single benchmark family, and no released weights for the two largest ESM-2 configurations. What it establishes is that a modest continued-pretraining budget is enough to retrofit long-context attention onto an existing protein language model without losing representation quality — and, in the quantized case, that 4-bit weights remain competitive on frozen-embedding transfer. The authors point to secondary structure and contact map prediction as the natural next fine-tuning targets, and to ProtT5 as the next architecture worth adapting the same way.