Loading…
Int4 LoRA adapters over long-context ESM-2 checkpoints, cutting the 33-layer load footprint to 664 MB and adding a 36-layer configuration.
No providers recorded yet. Browse all providers
The ESM-2 protein language models accept at most 1,022 amino acids, an architectural consequence of global self-attention whose cost grows with the square of sequence length. Raising that ceiling costs memory, and memory is the constraint that usually decides whether a lab can use a large protein language model at all.
ESM2 Quantized, from the Institute of Computing at the University of Campinas, attacks both problems at once. Gabriel Bianchin de Oliveira, Helio Pedrini, and Zanoni Dias re-pretrained ESM-2 with LongFormer-style local windowed attention over a 2,050-position context, as in the full-precision ESM2 Long family, but did it in 4-bit integer precision with LoRA adapters over companion base checkpoints. The 33-layer model loads in 664 MB against 3,338 MB for its full-precision counterpart.
The payoff shows up as coverage. Because quantized training is cheaper, this family reaches the 36-layer configuration that compute constraints put out of reach for the full-precision models — so the largest long-context checkpoint the project released is a quantized one. The work was presented at the XVII Brazilian Symposium on Bioinformatics (BSB 2024) and posted to arXiv in January 2025.
Each architecture keeps ESM-2's layer count, head count, and embedding width, changing only the attention pattern, the context length, and the numeric precision. Pretraining ran for five epochs over all 569,793 proteins in the July 2023 Swiss-Prot release, with AdamW at a learning rate of 1e-5, in int4 with bfloat16 compute.
Evaluation used protein function prediction on a CAFA5-derived benchmark, with roughly 74,000 training proteins and 500 Gene Ontology terms per sub-ontology. Last-layer embeddings were extracted with the backbone frozen and fed to an AutoKeras classifier selected over 50 trials, isolating representation quality. Across the whole test set the 33-layer model reaches an Fmax of 0.549 on Biological Process and 0.783 on Molecular Function, against 0.540 and 0.773 for standard ESM-2.
The result worth dwelling on is that quantization does not cost accuracy here — on several cells the quantized model is the better of the two long-context families, and on the subset of proteins longer than 1,024 amino acids the 33-layer quantized model reaches 0.556 on Biological Process against 0.517 for standard ESM-2. At these model sizes 4-bit weights remain competitive on frozen-embedding transfer, which is what makes the memory saving close to free.
The target user is anyone who wants long-context protein embeddings on hardware that will not hold a multi-gigabyte model. Function annotation, family assignment, and conservation analysis all consume ESM-2 embeddings and all pay a windowing penalty on the roughly 10–12% of proteins in a typical eukaryotic set that exceed 1,024 residues; this family removes that penalty without the memory bill. Anyone specifically needing the 36-layer configuration has no alternative within this project, since the full-precision family stops at 33 layers.
This is a focused engineering contribution, scoped to a single downstream task and a single benchmark family. What it establishes is narrower than a new modelling paradigm but useful: that 4-bit weights remain competitive on frozen-embedding transfer for protein language models, and that quantized training can extend a checkpoint family to sizes a full-precision budget could not reach. The practical caveat is distribution rather than method — the adapters are published as raw weight files with no modelling code implementing the local attention their config declares, and no repository declares a license, so reuse terms are undefined.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.