Antibody-specific BERT language model pretrained on 558M natural variable-domain sequences, and the sequence encoder behind IgFold.
No providers recorded yet. Browse all providers
A person's antibody repertoire is a record of an immune response in progress. After a B cell's receptor binds an antigen, the cell is expanded and its antibody gene is mutated over successive rounds, so a single repertoire sample holds germline-like ancestors alongside heavily mutated descendants. General protein language models learn from proteins shaped by many different selective pressures, whereas antibodies are selected almost entirely for binding. AntiBERTy is a language model trained only on antibody variable domains, so that its representations reflect the sequence statistics of V(D)J recombination and affinity maturation rather than those of the wider protein universe.
AntiBERTy was developed by Jeffrey Ruffolo, Jeffrey Gray and Jeremias Sulam at Johns Hopkins University and presented at the Machine Learning for Structural Biology workshop at NeurIPS 2021. It is a masked language model: residues in an antibody sequence are hidden and the model learns to recover them from context. The same pretrained checkpoint is then used without retraining to embed repertoires, to score sequences, and as the language-model component of the Gray lab's IgFold antibody structure predictor.
AntiBERTy is a different model from AntiBERTa, Alchemab's antibody language model of the same period, despite the near-identical name. The two share neither authors, code nor training set.
AntiBERTy uses the Hugging Face BERT encoder with 8 layers, 8 attention heads per layer, a hidden dimension of 512 and a feed-forward dimension of 2,048, for approximately 26M trainable parameters. Tokenization is one token per amino acid. Training data came from the Observed Antibody Space (OAS), which at the time held over 1B variable-domain sequences from 80 repertoire sequencing studies. Clustering at 95% identity with LinClust produced a non-redundant set of 588M sequences; 5% was held out for future testing, and the model was trained on 558M sequences with 1M reserved for evaluation. Training ran for 8 epochs on four NVIDIA A100 GPUs over roughly 10 days, using the masked language modeling objective.
The paper evaluates the embeddings on IgG repertoires from four donors who developed VRC01-class HIV-1 broadly neutralizing antibodies (40,517 to 77,067 sequences each). UMAP projections of AntiBERTy k-nearest-neighbour graphs showed continuous trajectories ordered by distance from germline, and three donors whose VRC01 lineages share ontogeny overlapped in the combined embedding. Multiple-instance-learning classifiers trained on these embeddings to detect highly redundant sequences reached AUROC 0.79 to 0.96 per donor and 0.84 on the combined repertoire; for 7 of 10 crystallized VRC01 antibodies, one attention head concentrated on the heavy-chain H2 loop and the following beta strand, matching the known VRC01 paratope.
AntiBERTy serves as a general-purpose antibody sequence encoder. Immunologists analysing repertoire sequencing data can embed a donor's sequences and order them by maturation state without building a lineage tree first. Antibody engineers use the pseudo log-likelihood as an antibody-likeness score for triaging designed or humanized candidates, and the species and CDR-graft heads to check a sequence's provenance. Because the encoder is small and ships as an MIT-licensed, pip-installable package, it is also a common frozen backbone for downstream predictors: structure prediction, heavy-light chain pairing and antigen-specificity classification all build on its per-residue features.
AntiBERTy's largest downstream role is as the language-model component of IgFold, the Gray lab's antibody structure predictor, which turns AntiBERTy embeddings into backbone atom coordinates through graph networks in about 23 seconds per paired antibody — a mean CDR H3 backbone RMSD of 3.27 Å against 3.56 Å for AlphaFold-Multimer, at a fraction of the runtime. Beyond that, it has become one of the standard antibody-specific baselines: catalog entries including Ab-RoBERTa, PARA and AntiBERTa2 benchmark against it directly, and it is a recurring choice of pretrained backbone in antibody fine-tuning studies. Its limitations are equally clear from later work: it is a small model trained on unpaired chains, it predates the paired-chain and larger antibody encoders that followed, and the weakly supervised paratope results are a proof of concept on one antibody class rather than a validated binding-site predictor. The paper remains a workshop preprint and has not been peer reviewed.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.