AbLingua

A protein language model reading an antibody sees the letter S as one symbol whether it sits in a rigid framework β-strand or in the crown of CDR H3. The backbone dihedral angles that decide which of those it is are set largely by the residue's immediate neighbors, so a twenty-letter alphabet asks the model to reconstruct local geometry from tokens that carry none of it. The usual answer is to add parameters and hope the attention layers recover the missing context — and antibody language models have repeatedly reported that scaling up buys less than it should.

AbLingua, from the International Digital Economy Academy in Shenzhen, changes the token instead of the architecture. Its TripleAA scheme slides a window of three residues along the chain, so every position becomes one token carrying that residue plus the one on either side: ABCDE tokenizes to ABC, BCD, CDE. Built from 26 letters and two boundary symbols, the vocabulary holds 21,952 entries — the scale of a natural-language vocabulary rather than a protein alphabet — while preserving a one-token-per-residue alignment that byte-pair encoding destroys. With that encoding in place, the family scales cleanly from 300M to 1.7B parameters, with perplexity falling on a power law in both model size and data volume.

The 1.7B member is pretrained on 1.4 billion antibody sequences, which the authors describe as the largest encoder-based antibody-specific language model reported. It sits alongside AbLang-2, AntiBERTa2 and general-purpose ESM-2 as a representation backbone, and shares its lab with ImmuFold, IDEA's antibody structure predictor.

#Key Features

• Tripeptide tokenization: Each token spans a residue and its two neighbors, encoding the local cooperative interactions that determine β-turns and loop conformations without any explicit structural input.
• Context residue transformation: A masking scheme built for overlapping tokens. Joint masking hides all three tokens containing a target residue so its identity cannot leak from a neighbor, and joint replacement substitutes the residue consistently across all three.
• CDR-weighted pretraining: Tokens in complementarity-determining regions are masked at 30% against the standard 15% elsewhere, concentrating the objective on the hypervariable segments.
• Framework paratope prediction: Contact residues outside the CDRs are sparse and usually ignored; the fine-tuned 1.7B model reaches 62% precision on framework paratopes and 76% on CDR paratopes.
• Label-free repertoire structure: Embeddings separate five B-cell developmental stages and HIV-, SARS- and Ebola-specific antibodies under UMAP with no supervision.

#Technical Details

AbLingua is a BERT encoder with relative positional embeddings and a 256-token context, enough for a single antibody chain. Pretraining data came from 2.4 billion unpaired Observed Antibody Space sequences clustered with MMseqs2 at 0.9 identity, yielding OAS-Unpaired-1.4B — 1.2 billion heavy and 200 million light chains, shuffled together. The 300M and 600M variants train on a 300M-sequence subset. On unified paratope prediction the fine-tuned 1.7B model reaches roughly 72% recall and 75% F1, beating proABC-2 despite using no multiple sequence alignments. Paired with a ResNet head on HER2 CDRH3 screening data it achieves 58.0% validation recall against 57.0% for ESM2 and 56.3% for IgBert, and 85% recall on an independent set of sequences validated by bio-layer interferometry. The publicly released checkpoint is the 600M variant: 30 layers, hidden size 1280, 20 attention heads, an 18,950-token vocabulary and roughly 617M parameters, distributed under Apache-2.0. The 300M and 1.7B checkpoints are retained by IDEA.

#Applications

The model is used as a frozen feature extractor or a fine-tuning backbone for antibody engineering: predicting which residues form the paratope, scoring neutralizing capacity directly from sequence, and ranking CDRH3 variants from large screening libraries. That last case is the most immediately practical — the HER2 experiment shows a predictor trained on noisy FACS-plus-deep-sequencing labels can still recover genuine binders, which shifts effort away from high-precision affinity assays during lead optimization. Immunologists working with repertoire sequencing get a second use: embeddings that organize by developmental stage and antigen specificity without labels.

#Impact

AbLingua's contribution is evidence that the ceiling earlier antibody language models hit was a property of character-level encoding rather than of antibody data being intrinsically low-information. Once tokens carry local sequence context, the familiar power-law relationship between scale and perplexity reappears, which gives the subfield a reason to build larger models and a way to predict what they will buy. The limitations are stated plainly by the authors: the encoder architecture cannot generate sequences the way decoder models such as ProGen2 and IgLM do, tokenization captures structural signal only implicitly, and training on unpaired chains within a 256-token window leaves paired heavy–light modeling outside the released model's range. The gap between the 1.7B model the paper benchmarks and the 600M checkpoint the public can download is the practical constraint on reproducing these results.