AbCDR-ESM2 / AbCDR-ESMC

Only about a fifth of an antibody variable domain does the recognizing: the six complementarity-determining regions (CDRs) form the loops that touch antigen, while the framework regions hold them in place. A protein language model trained the conventional way — mask 15% of residues uniformly — spends most of its supervision on that conserved scaffold, where the next residue is easy to guess, and little on the loops whose sequence decides whether an antibody binds. The base ESM2-3B checkpoint shows the consequence: it recovers 72–92% of masked framework residues but only 35.7% of heavy-chain CDR3.

AbCDR is the region-aware answer: 50% of residues inside annotated CDRs are masked during adaptation, with the framework left intact to supply context, so the training signal concentrates where antibody function lives. The work comes from Boston University — engineering, chemistry, and the National Emerging Infectious Diseases Laboratories — with the Oden Institute at the University of Texas at Austin, funded by Merck Research Laboratories, the NSF, and the Department of Energy. Published in Communications AI & Computing after an October 2025 bioRxiv preprint, the paper names two checkpoints: AbCDR-ESM2, adapted from ESM-2 at 3B parameters, and AbCDR-ESMC, adapted from ESM Cambrian at 600M.

Its most useful result is a negative one. The authors also pretrained on 1.22 billion unpaired OAS sequences, more than 300 A100 GPU-hours, and found it gave no measurable benefit once paired fine-tuning followed. Paired data with CDR-aware supervision, not repertoire scale, moved the numbers.

#Key Features

• CDR-focused masking: Half of the residues within annotated CDRs are masked, following the BERT 80/10/10 corruption pattern; the 50% rate was set so the masked-token count matches conventional 15% whole-chain masking.
• Paired variable domains: Heavy and light chains are concatenated with a dedicated separator token, so attention spans the VH–VL interface instead of treating each chain as an isolated protein.
• Two backbone scales, one recipe: A 3B and a 600M adaptation share the masking scheme, and the compact model matches or exceeds antibody-specific baselines several times its size.
• Frozen embeddings with a linear probe: Mean-pooled representations feed L2-regularized ridge regression onto log dissociation constants; the language model is never fine-tuned on binding labels.
• Pareto-selected checkpoints: The released epoch is the earliest one maximizing CDR recovery while framework accuracy drops no more than 0.1 percentage points, guarding against forgetting the scaffold.

#Technical Details

Adaptation used 1,617,948 cluster representatives from the Observed Antibody Space, clustered at 95% identity with MMseqs2, holding out 20,225 sequences each for validation and test. ESM2-3B follows a two-stage curriculum — whole-chain masking at 15% for five epochs, then CDR-only masking at 50% for three — while ESM C reaches its selected checkpoint with direct CDR masking at epoch four. Training ran on 16 A100 GPUs in bfloat16 at a global batch of 256 with AdamW, using Hugging Face Transformers, Accelerate, and DeepSpeed ZeRO. Embeddings are 2560-dimensional for ESM2 and 1152 for ESM C.

CDR-focused adaptation lifts heavy-chain CDR3 recovery from 35.7% to 62.9% with framework accuracy essentially unchanged. On binding data drawn from FLAb — single-mutant panels against VEGF (n = 2048), hen egg-white lysozyme (n = 4275), and HER2 (n = 422), plus combinatorial libraries against a SARS-CoV-2 spike HR2 peptide (n = 71,830), fluorescein, and H1 hemagglutinin — AbCDR-ESM2 raises R² on VEGF from 0.302 to 0.359 over its base model and reaches 0.396 on the SARS-CoV-2 set, 8.8% above the strongest published baseline. The largest gain is on fluorescein, where R² climbs from 0.547 to 0.693. Improvements run to 27%, and the adaptation takes eight epochs against 46 reported for IgBERT. Masked-recovery comparisons against IgBERT and IgT5 use different test splits and are indicative only.

#Applications

The model suits antibody discovery and affinity maturation, where a team holds binding measurements for a few thousand variants and wants to rank the rest before committing them to expression and characterization. Because the backbone stays frozen, adopting it for a new antigen means extracting embeddings once and fitting a ridge head on whatever labels exist — no GPU fine-tuning per campaign, and the 600M checkpoint keeps that within reach of modest hardware. Baselines such as AbLang, IgBERT, and IgT5 remain the natural points of comparison.

#Impact

Code, weights, and data are public: checkpoints and repository under MIT, both curated datasets under CC BY 4.0, with Zenodo archives of the versions behind the paper. The lasting contribution is methodological: where the masking budget is spent matters more than how many sequences it is spent on, which cuts the compute bill for antibody-specific adaptation. The stated limitations are real: only ESM-family backbones were tested, so transfer to architectures with different positional encodings is unexplored, and the evaluation covers binding affinity alone, leaving developability properties such as stability, solubility, and immunogenicity untouched.