Ab-RoBERTa

Antibody-specific language models are now a crowded field, but the useful ones are surprisingly hard to obtain. Several of the strongest are built on the RoBERTa architecture — a BERT variant that swaps static for dynamic token masking and drops the next-sentence-prediction objective — yet their pretrained backbone weights have stayed out of reach. AntiBERTa never released its foundational checkpoint; AbLang and Sapiens distribute only task-specific fine-tuned parameters. Researchers wanting a RoBERTa-style antibody encoder to build on have had to train one themselves.

Ab-RoBERTa, released in June 2025 by the Mogam Institute for Biomedical Research in South Korea, exists to close that gap. It is a 125M-parameter masked language model pretrained from scratch on 402 million human antibody variable-region sequences from the Observed Antibody Space (OAS) database, with weights published openly on HuggingFace under an MIT license. The design bet is on efficiency rather than scale: at 125M parameters it is roughly a third the size of the BERT-based ProtBERT and IgBERT and about a twenty-fourth the size of the 3B-parameter T5 models IgT5 and ProtT5, which makes fine-tuning and deployment materially cheaper.

The paper also settles a practical question that protein language modeling has largely taken on faith. The authors compare three tokenization schemes — single amino acid, dipeptide, and byte-pair encoding — and find that only single-amino-acid tokenization produces embeddings that separate B cell subtypes and target antigen classes, not just germline V gene families. That result motivates the released model's design.

#Key Features

• Openly released pretrained backbone: Unlike the RoBERTa-based antibody models that preceded it, the foundational checkpoint itself is published, so downstream groups can fine-tune or extract frozen embeddings without retraining.
• Human-repertoire pretraining corpus: Two billion OAS sequences were filtered to human variable regions with intact framework 1 and framework 4 segments, leaving 574 million sequences split into 402 million for training and 86 million each for validation and testing.
• Tokenizer ablation: Single-amino-acid, dipeptide, and byte-pair tokenizers were each pretrained on 40 million sequences and compared by UMAP structure of the resulting embeddings; single-amino-acid tokenization was the only scheme to resolve B cell subtype and antigen class.
• Efficiency-first sizing: At 125M parameters the model fine-tunes in roughly three hours on the paper's downstream tasks, against about sixteen hours for the 3B-parameter T5 models, while landing within a percentage point of their accuracy.
• Standard Transformers interface: The checkpoint loads directly through the HuggingFace Transformers library for feature extraction or fine-tuning, with the caveat that inputs must be uppercase amino acid strings with no interior spaces.

#Technical Details

Ab-RoBERTa uses the stock RoBERTa configuration — hidden size 768, 12 layers, 12 attention heads, 3072-dimensional feedforward — with a 25-token single-amino-acid vocabulary and a maximum position embedding length of 150 chosen to minimize padding on variable-region sequences. Pretraining used the standard masked-language-modeling objective at a 15% masking rate on three NVIDIA A100 GPUs for six epochs, roughly 654 hours, with AdamW at a 1e-4 peak learning rate and 30,000 warmup steps. Downstream evaluation covers three fine-tuned classification tasks: five-way target antigen prediction, B cell subtype prediction, and germline V gene family prediction, each averaged over five random seeds. On heavy-chain antigen classification Ab-RoBERTa reaches 0.850 AUROC, second only to IgT5 at 0.858 and ahead of AntiBERTy (0.837), IgBERT (0.823), and ProtBERT from the ProtTrans suite (0.786). On heavy-chain B cell type classification it is the best model tested at 0.890 AUROC, narrowly ahead of IgT5's 0.883. Germline V gene classification saturated at 1.0 for every model after a single epoch, which the authors read as evidence that the task is driven by raw sequence similarity rather than learned context.

#Applications

The intended uses are the standard antibody-engineering workload: paratope prediction, humanness assessment, and repertoire-scale analysis of B cell receptor sequencing data, either by fine-tuning the checkpoint or by using its embeddings as features. The efficiency argument matters most for immune repertoire studies, where millions of sequences must be embedded, and for groups iterating on many downstream heads where a five-fold reduction in fine-tuning time compounds quickly.

#Impact

Ab-RoBERTa's main contribution is availability — a competitive, permissively licensed antibody backbone in a subfield where the strongest models have often been described but not distributed — together with a concrete demonstration that single-amino-acid tokenization is the right default for antibody sequences. Its limitations should be read alongside that. Benchmarking is confined to three repertoire-annotation classification tasks; paratope prediction and humanness scoring, the applications the paper names, are not themselves evaluated. The work is a preprint that has not been peer reviewed, no training or evaluation code accompanies the weights, and the HuggingFace model card documents configuration and usage but there is no data card describing the filtered OAS training corpus.