Antibody language model pretrained on 1.3M natively paired heavy/light chains, learning cross-chain mutation patterns that unpaired models miss.
No providers recorded yet. Browse all providers
An antibody's binding site is built from six loops, three on the heavy chain and three on the light chain, and the two chains are chosen and mutated together inside a single B cell. Most antibody language models have nonetheless been trained on one chain at a time, because unpaired repertoire sequencing is cheap and paired sequencing is not. A model that never sees a heavy chain beside its native light partner cannot learn that somatic hypermutation in one chain predicts mutation in the other, or which light chains a given heavy chain tends to accept.
BALM-paired (Baseline Antibody Language Model, paired), from Sarah Burbach and Bryan Briney at Scripps Research, tests how much that missing context matters. It is a RoBERTa-large masked language model trained on the natively paired human antibody dataset of Jaffe et al., the largest public collection of its kind. The study is built as a controlled experiment: BALM-unpaired sees the same sequences with pairing removed, BALM-shuffled sees them with heavy and light chains randomly re-paired, and ft-ESM continues training ESM-2 (650M) on the same native pairs. The preprint appeared on arXiv in August 2023 and the peer-reviewed version was published in Patterns in May 2024.
</s> separator
into a single sequence of up to 512 tokens, so every attention layer can relate positions
across the VH/VL pair.BALM-paired uses a modified RoBERTa-large encoder with 24 layers, 16 attention heads, a hidden size of 1,024 and a feed-forward size of 4,096, giving about 304 million parameters, with a 25-token vocabulary of the 20 amino acids plus five special tokens. The Jaffe dataset was annotated with abstar and filtered for duplicates and non-productive sequences, leaving 1,335,854 heavy/light pairs from circulating B cells of healthy adult donors, split 90/5/5 for training, evaluation and testing. Training used a standard masked-language-model objective (15% of tokens selected with dynamic masking) for 500,000 steps, about 100 epochs, at a batch size of 256 and a peak learning rate of 4e-4 on eight NVIDIA A100 GPUs, taking roughly five days. BALM-unpaired used a batch size of 512 so both models took the same number of steps. ft-ESM was trained on the same pairs for 150,000 steps.
For downstream evaluation, each model was fine-tuned with a classification head on three antigen-specificity tasks built from CoV-AbDab, a curated influenza antibody set and healthy-donor memory B-cell repertoires: coronavirus versus healthy donor (about 18,000 sequences), coronavirus versus influenza (about 2,500) and a three-way task (about 3,700). The paired models beat their unpaired counterparts, BALM-unpaired and base ESM-2, on all three. ft-ESM led on the largest binary task, and BALM-paired, at under half its parameter count, matched or exceeded it on several metrics of the smaller tasks. BALM-shuffled did well on the binary tasks but fell back toward the unpaired models on the three-way task, so part of the paired advantage comes from familiarity with the two-chain input format and part from native cross-chain features.
BALM-paired is a general-purpose encoder for paired B-cell receptor sequences from single-cell repertoire sequencing. Its embeddings and fine-tuned heads support antigen specificity classification, repertoire clustering, and masked-residue prediction to explore which somatic mutations are plausible given a partner chain. Antibody discovery teams can use it as a starting checkpoint for paired-sequence property predictors. The model is sequence-only: it does not predict structure or binding affinity, and all evaluation is in silico.
BALM-paired gave the field a controlled measurement of what native pairing is worth, at a time when AntiBERTa and AbLang had been trained on single chains, and it showed that fine-tuning a general protein language model on a modest paired set recovers much of the same cross-chain signal. The same group built on the result with CurrAb, which uses a curriculum to combine scarce paired data with abundant unpaired sequences. It is unrelated to the binding-affinity model BALM from the University of Edinburgh, which shares the acronym. Training code is MIT-licensed on GitHub, and the Zenodo deposit holds weights for all four models alongside the training splits. The checkpoints are also on Hugging Face.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.