AntiBERTa2

Two antibodies whose CDRH3 loops differ by a single residue can fold into different shapes and bind different antigens; two sharing almost no sequence identity can converge on the same loop conformation. That breaks the assumption most protein language models rest on, that sequence neighbourhood is a usable proxy for structural neighbourhood. Antibodies concentrate nearly all of their variation into a few short hypervariable stretches, and a model trained across the whole protein universe spends most of its capacity elsewhere.

AntiBERTa2 is Alchemab's response: an encoder trained only on human immunoglobulin sequences, and on enough of them that the grammar of the complementarity-determining regions is learned from natural repertoires rather than inferred from general protein statistics. It is a RoFormer — a bidirectional transformer whose positional signal comes from rotary embeddings rather than learned absolute position vectors — pretrained by masked language modelling on 779.4 million human antibody sequences. It succeeds AntiBERTa, the 2022 BERT-style model from the same group, with roughly fourteen times the pretraining data.

The substantive change is chain pairing. AntiBERTa encoded heavy and light chains as separate, unlinked sequences, so it could not represent the combined binding site the two chains form. AntiBERTa2 accepts a heavy chain, a light chain, or both joined by a [SEP] token, with a single-token chain marker prefixing each chain, so one encoder serves bulk unpaired repertoire data and paired single-cell data. The work appeared as a NeurIPS 2023 Machine Learning for Structural Biology workshop paper and reached bioRxiv in January 2024.

#Key Features

• One encoder for paired and unpaired repertoires: Chain-marker tokens and a [SEP] join let the same weights embed heavy-only, light-only, and paired heavy–light inputs, matching both formats antibody sequencing produces.
• Rotary position embeddings: Position enters as a rotation of the query and key vectors, encoding relative rather than absolute offsets — a better fit for CDR loops whose length varies between antibodies.
• Repertoire-scale, human-only pretraining: The corpus is drawn entirely from human B cell receptors, so masked infilling proposes residues that natural human repertoires actually use.
• Masked infilling as a design primitive: Run as a standard fill-mask pipeline over masked CDR positions, the model generates candidate variants directly, with no task-specific head.
• Transfers with the encoder frozen: Binding prediction was evaluated with all encoder weights fixed and only a classification head trained, so the representations carry the signal on their own.

#Technical Details

The released checkpoint is a 202-million-parameter RoFormerForMaskedLM: 16 layers, 16 attention heads, 1024-dimensional embeddings, 4096-dimensional feed-forward width, a 30-token vocabulary, and a 256-position context — enough for a paired Fv, but not for longer constructs. Pretraining ran 260,000 steps with 40,000 warmup under AdamW at a peak learning rate of 8e-5, on 48 A100 GPUs of NVIDIA's Cambridge-1 system.

The corpus began with 1.47 billion unpaired human antibody sequences pulled from the Observed Antibody Space in February 2023 plus 70 million proprietary sequences, clustered with Linclust at 90% identity down to 821.2 million. Paired data — 1.5 million public plus 1.4 million in-house — was deduplicated at 99% identity to 2.5 million, and the combined pool was split 95:5 into 779.4 million training and 44.3 million evaluation sequences. On the Mason et al. HER2 benchmark of 39,108 trastuzumab CDR variants (18,223 training, 2,278 test), a frozen encoder with a trained classification head reaches AUROC 0.839, AUPR 0.827 and F1 0.754, ahead of AntiBERTy at 0.663 and ESM-2 650M at 0.781. Under ablation it holds AUROC 0.590 with only 19 labelled training sequences.

#Applications

The model suits antibody discovery and engineering groups working from repertoire sequencing. Masked infilling over CDR positions proposes variants for a design campaign; per-sequence embeddings feed lightweight classifiers for antigen binding, developability, or humanness without fine-tuning the backbone; and paired-chain support makes it usable directly on single-cell BCR data. It loads through transformers as an ordinary RoFormer checkpoint. The licence is the practical constraint: weights carry a modified Apache 2.0 grant limited to non-commercial research, and the restriction extends to antibodies generated by the model, so therapeutic and diagnostic programmes need a separate arrangement with Alchemab.

#Impact

AntiBERTa2 is the checkpoint Alchemab built on next: their contrastively trained variant, AntiBERTa2-CSSP, starts from this encoder and aligns it against a frozen structure encoder, and that variant is in turn the backbone of ImmunoMatch. The same group later scaled the idea further in FAbCon. Its limits are worth stating plainly: no training or fine-tuning code was released with the weights, part of the pretraining corpus is proprietary and cannot be reconstructed from public sources, the reported evaluation covers structural-similarity regression and a single antigen, and the workshop paper has not been through journal peer review. What it establishes is that antibody-specific pretraining at repertoire scale yields representations carrying binding signal in the low-data regime where antibody engineering usually operates.