AntiBERTa2-CSSP

One substitution in a CDRH3 loop can change an antibody's shape and its target; two antibodies sharing almost no sequence identity can fold that loop the same way. A language model trained on antibody sequence alone inherits that mismatch — its embedding space is organised by sequence neighbourhood, which is a poor proxy for structural neighbourhood exactly where binding is decided. Multimodal protein models fix this by taking structure as a second input, but that only moves the problem: a repertoire sample holds around 100,000 sequences, and even a one-second-per-structure predictor would need over a day of compute per donor.

AntiBERTa2-CSSP is Alchemab's way around that trade-off — structure enters during training and never at inference. It starts from AntiBERTa2, the group's RoFormer antibody encoder, and continues pretraining it against a frozen structure encoder under a contrastive objective borrowed from CLIP: sequence embeddings play the role of "text", structure embeddings the role of "image", and the two are pulled into a shared latent space by a temperature-scaled cosine similarity loss. Where earlier sequence–structure work updated both towers and kept both at inference, here the updated sequence tower is detached and shipped alone, consuming an amino acid string and nothing else.

The work appeared at the NeurIPS 2023 Machine Learning for Structural Biology workshop, with a bioRxiv version in January 2024. CSSP is backbone-agnostic — it also lifts AntiBERTy and ESM-2 650M — but AntiBERTa2 is the arm Alchemab optimised and released.

#Key Features

• Structure supervises training, sequence drives inference: The contrastive stage needs paired sequences and coordinates, but the deployed encoder needs neither a crystal structure nor a predicted one, so it runs at repertoire scale.
• CLIP objective over antibody sequence–structure pairs: [CLS] pooling on the sequence side and average-pooled embeddings from a frozen inverse-folding encoder on the structure side are projected into one 512-dimensional space and matched within each batch.
• Embeddings that follow backbone geometry: A regression head over paired [CLS] embeddings recovers structural similarity at Pearson r 0.706 for whole-Fv RMSD and 0.576 for CDRH3 loop RMSD, the highest of the three CSSP arms.
• Data efficiency in the low-label regime: On HER2 binding it reaches AUROC 0.817 from 912 labelled antibodies, an eight-fold reduction in labels needed.
• A drop-in transformers encoder: Loads as RoFormerModel for embeddings or RoFormerForSequenceClassification for a fresh head.

#Technical Details

The released checkpoint is a 16-layer, 16-head RoFormer with 1024-dimensional embeddings, a 4096-dimensional feed-forward width, a 30-token vocabulary and a 256-position context — 201.6 million parameters in float32, declared as a bare RoFormerModel, so the base model’s masked-language head is not carried over. Contrastive training used a 512-dimensional joint embedding, dropout 0.3, a temperature clamp of 100, peak learning rate 1e-3 under AdamW with weight decay 0.25, for 10 epochs; HER2 AUROC plateaus after about five.

The structural corpus is 1,554 human antibody structures from 995 PDB entries, pulled from SAbDab in July 2023 and filtered to X-ray or cryo-EM at 2.5 Å or better with no single-chain variable fragments and no missing CDR coordinates, split by PDB code into 1,237 training, 155 validation and 162 test structures. The sequence side inherits AntiBERTa2's 779.4-million-sequence masked-language-modelling corpus. Benchmarked on 39,108 trastuzumab CDR variants screened against HER2 (18,223 training, 2,278 validation, 2,278 test) with the encoder frozen and only a classification head trained, it reaches AUROC 0.875, AUPR 0.869 and F1 0.807, against 0.839, 0.827 and 0.754 before CSSP; the structural training set contains no trastuzumab relatives. Predicted structures proved a poor substitute — 12,370 ABodyBuilder2 models trained a weaker encoder than 1,237 crystal structures, and mixing the two gave no synergy.

#Applications

The checkpoint suits antibody discovery groups needing an encoder that carries binding signal when labels are scarce: freeze it, train a light classifier over the pooled embedding, and rank candidates for antigen binding. Because inference is sequence-only, it applies to whole repertoire datasets rather than the handful one could realistically fold, and its embedding space supports structure-aware clustering where sequence identity misleads. The licence is the operative constraint: a modified Apache 2.0 grant limited to non-commercial research, barring pre-clinical and clinical use and extending to antibodies generated with the model; commercial terms are arranged separately with Alchemab.

#Impact

The result that matters beyond antibodies is that barely a thousand experimental structures measurably reshape a language model trained on hundreds of millions of sequences, and that information content, not volume, carries the gain. It has been picked up as a backbone in its own right: ImmunoMatch fine-tunes it to predict heavy–light chain pairing. Its limits are worth stating plainly: no CSSP training code was released, part of the inherited pretraining corpus is proprietary, the reported evaluation covers one structural-similarity regression and a single antigen, and the workshop paper has not been peer reviewed.