Loading…
Scaling-law study of protein language models identifying compute-optimal training for causal and masked objectives on 939 million protein sequences.
Compute-Optimal PLM — released by its authors under the name ScalingProteinLM — is a systematic study from BioMap Research, Tsinghua University and MBZUAI that applies Chinchilla-style scaling law analysis to protein language models. It first appeared on bioRxiv in June 2024, was posted to arXiv as 2411.02142, and was accepted as a spotlight paper at NeurIPS 2024. The work addresses a practical gap: protein language models had grown rapidly in scale, from BERT-sized models to 10-billion-parameter architectures, with little principled guidance on how to split a fixed compute budget between model size and training data volume.
The central question the authors pose is whether existing protein language models, including ESM-2 and ProGen2, are actually trained at their compute-optimal operating point. Their empirical answer is that most are not. By training over 300 models from 3.5 million to 10.7 billion parameters on 5 to 200 billion unique tokens, the team derives scaling laws for causal (CLM) and masked (MLM) language modeling objectives specific to the statistical structure of protein sequence data.
The study incorporates 939 million protein sequences, drawing on metagenomic sequences in addition to the standard UniRef database. This deliberate expansion of data diversity directly addresses two failure modes the authors identified: diminishing returns from repeated tokens under CLM training, and overfitting to UniRef under MLM training.
The study trains encoder-only (MLM) and decoder-only (CLM) transformer architectures across a grid of sizes from 3.5M to 10.7B parameters, each trained on between 5B and 200B unique protein sequence tokens. The corpus combines UniRef with metagenomic sequences to reach 939 million total. Following the Chinchilla methodology, the authors fit parametric loss curves of the form L(N, D) = A/N^alpha + B/D^beta + E, where N is parameter count and D is token count, and identify the optimal (N, D) frontier for a given compute budget C = 6ND.
Protein CLMs exhibit stronger diminishing returns with repeated data than NLP models, so data diversity matters more than data repetition in this modality. The optimal compute allocation for protein MLMs favors larger datasets relative to model size compared to equivalent NLP budgets. The two compute-optimal checkpoints the study selected — a 7B CLM and a 10B MLM — are distributed as proteinglm-7b-clm and proteinglm-10b-mlm on the ProteinGLM HuggingFace hub, the same channel that carries xTrimoPGLM from the same lab, under a non-commercial CC BY-NC 4.0 license. Downstream evaluations on protein generation quality and structure and function prediction benchmarks confirm that these compute-optimal models match or exceed ESM-2 and ProGen2 at equivalent or lower pretraining compute costs.
The primary audience is research teams planning to train protein language models from scratch or continue pretraining existing ones. Given a compute budget in total FLOPs, the laws fix the tradeoff between model size and training tokens in advance, heading off the common failure of an over-parameterized model trained on too little data. The frontier also guides model selection: practitioners choosing among off-the-shelf checkpoints can see which one is best matched to its computational cost. The released 7B CLM and 10B MLM checkpoints are directly usable for downstream protein sequence tasks including fitness prediction, structure-guided generation, and functional annotation.
By transporting Chinchilla-style reasoning into the protein domain, the study set a methodological precedent for compute-optimal training that extends naturally to RNA and genomic sequence models. A key limitation acknowledged by the authors is that the scaling laws are derived from transformer-based CLM and MLM architectures trained on sequence data alone; they do not extend directly to structure-conditioned models or multimodal architectures that jointly process sequence and structure. The finding that existing large protein language models are often not compute-optimal implies the field has underinvested in data diversity relative to model scale — a correction that lowers the barrier for groups with modest compute resources to train competitive models.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.