AbLang-2

Mask a residue in an affinity-matured therapeutic antibody and ask an antibody language model what belongs there. It answers with the germline residue — even at positions where somatic hypermutation put something else, and even though those departures from germline are what gave the antibody its affinity. Measured on masked non-germline (NGL) residues, Sapiens returns the germline 87.6% of the time, AntiBERTy 86.7%, and AbLang 84.9%. The models are not confused; they are correct about the training distribution and unhelpful for the design question being asked of them.

The bias comes from the data. Paired antibodies in the Observed Antibody Space (OAS) come predominantly from naive B cells (42%) and unsorted B cells (39%), with only 17% from memory B cells, so most of the variable domain in most training sequences is unmutated germline. AbLang-2, from the Oxford Protein Informatics Group and GSK, treats this as a class-imbalance problem rather than a data-volume problem. The contribution is a training recipe aimed at the rare residues: focal loss in place of cross-entropy, a modified masking scheme, and pretraining on unpaired chains followed by fine-tuning on paired VH-VL antibodies.

That framing separates it from its predecessor. AbLang trained separate heavy and light chain models for restoring truncated sequences; AbLang-2 is one model that reads both chains together and is optimised for proposing mutations, on an ESM-2-style backbone rather than AbLang's RoBERTa one.

#Key Features

• Focal loss on rare residues: Down-weighting the loss on easy, well-predicted germline positions redirects gradient onto NGL residues. This single change gave the largest gain in the ablation, cutting NGL perplexity from the 14.23–38.95 range to 10.24–12.69 without degrading germline accuracy.
• Paired VH-VL input: Heavy and light chains are concatenated with a separator token, so one checkpoint handles unpaired VH, unpaired VL, and paired antibodies through the same forward pass.
• Mixed masking regime: Each batch draws from random masking, short 3–5 residue spans, or a single long span, with the masked proportion sampled uniformly between 10% and 40% rather than fixed at 15%.
• Diverse valid suggestions: On clonotype positions with known NGL alternatives, AbLang-2 places 15% (VH) and 14% (VL) cumulative probability on those residues while keeping over 90% across germline plus NGL — it proposes alternatives without proposing implausible amino acids.
• Split representation and likelihood heads: AbRep emits 480-dimensional per-residue embeddings, AbHead converts them to per-position amino acid likelihoods. The same architecture, loaded with separate weights, serves paired T-cell receptors as TCRLang-Paired.

#Technical Details

AbLang-2 is a 12-block bidirectional transformer with a 480-dimensional embedding, 20 attention heads, SwiGLU activations, rotary position embeddings, and a 26-token vocabulary — roughly 44 million parameters. Training data came from a November 2022 OAS snapshot of 2,072M VH, 357M VL, and 1.57M paired sequences, filtered for duplicates, missing cysteines, and fragments, then redundancy-reduced with Linclust at 95% identity to 35.6M unpaired chains and 1.26M paired antibodies. The model was pretrained for 200,000 steps on unpaired sequences and fine-tuned for 10,000 steps on paired ones, using Adam with a 1,000-step warm-up, a peak learning rate of 0.0004, cosine decay, weight decay of 0.01, and an effective batch of 8,192 sequences.

Evaluation splits perplexity by residue class, because whole-sequence perplexity is dominated by germline positions and hides the failure mode. On NGL residues, AbLang-2 reaches perplexities of 9.92 (heavy framework), 11.13 (heavy CDR1/2), and 12.47 (heavy CDR3), against AbLang's 25.80, 17.73, and 14.47; on light chains, 10.09, 9.54, and 10.77 against 52.14, 25.72, and 16.75. Germline perplexity stays near 1 throughout. A second evaluation grouped 100k held-out paired antibodies into 101 clonotypes, giving 226 VH and 60 VL sites with at least three observed NGL residues.

#Applications

The model is built for in-silico antibody optimisation: masking framework or CDR1/2 positions in a lead candidate and reading off substitutions that natural affinity maturation would plausibly produce. AbRep embeddings feed downstream developability, immunogenicity, and binding predictors, and the paired input matches BCR-seq protocols that preserve VH-VL pairing. The germline-bias analysis was benchmarked against 735 paired therapeutic antibodies drawn from Thera-SAbDab, so the evaluation targets the molecules an antibody engineering group actually works on. It ships as a pip-installable Python package with optional ANARCI-based numbering.

#Impact

AbLang-2's more durable contribution is methodological: it showed that reporting whole-sequence perplexity for antibody language models conceals near-random performance on exactly the residues that matter for design, and that the fix is a loss function rather than more sequences. Several limitations are stated plainly. Suggesting mutations from an unmasked, germline-reverted sequence in a single forward pass remains worse than random, so residue-by-residue masking is still required. The cumulative-probability metric counts only mutations already observed in the clonotype set and weights them equally regardless of functional importance. All results are computational; no wet-lab validation of suggested mutations is reported.