CurrAb

Antibody language models face a data problem that general protein language models do not. The sequences that matter most — natively paired heavy and light chains from the same B cell, which together define a binding site — exist in the low millions, while unpaired heavy or light chain sequences number in the hundreds of millions. Models trained only on paired data are starved of scale; models trained only on unpaired data never see the interchain context that shapes a paratope. Mixing the two is the obvious answer, but the two-orders-of-magnitude imbalance means the mixing strategy itself decides whether the combination helps.

CurrAb, from Sarah Burbach and Bryan Briney at Scripps Research, is a 650M-parameter antibody language model that resolves this with curriculum learning. Rather than fixing the paired/unpaired ratio or pretraining on unpaired data and then fine-tuning on paired data, CurrAb samples from a probability curve that begins heavily weighted toward unpaired sequences and slides toward paired sequences as training proceeds. The accompanying study systematically compares this curriculum against a constant mix and a fine-tuning schedule across model scales, making it as much a methods evaluation as a model release. It appeared as a bioRxiv preprint in March 2025 and was published in PLOS Computational Biology in September 2025.

#Key Features

• Sigmoid curriculum schedule: The probability of drawing an unpaired sequence follows a sigmoid decay from 0.7 to 0.3 across training, with a slope parameter tuned by sweep. At no point does either data type fall below 20% of a batch.
• Paired chains as a single sequence: Heavy and light chains are concatenated with a separator token, so self-attention spans the interchain interface rather than treating the two chains as independent inputs.
• Direct comparison of mixing strategies: Curriculum, constant-mix, and fine-tuned 650M checkpoints were all trained and released, isolating the effect of the schedule rather than confounding it with architecture or data changes.
• Repertoire-scale downstream tasks: Evaluation covers masked-token loss on memory B-cell sequences, native-versus-shuffled chain pairing, and three-way antigen specificity classification.

#Technical Details

CurrAb is a masked language model built on a modified ESM-2 encoder with rotary position embeddings, a 33-token vocabulary, and inputs padded to 320 positions. Pretraining ran for 500,000 steps at a batch size of 512, with a peak learning rate of 1e-4 after 30,000 warmup steps and standard 15% masking. The corpus combines 151,764,423 unpaired and 1,717,423 paired antibody sequences drawn from the Observed Antibody Space database plus roughly 400,000 internally generated paired sequences, all clustered at 90% identity and split 96/4 for training and evaluation.

On held-out memory B-cell repertoires, CurrAb records the lowest masked-token cross-entropy of the antibody language models evaluated — IgBERT, AbLang2, and AntiBERTa2 — on both paired sequences and unpaired heavy chains, with the largest margin in the hypervariable CDRH3 region. On three-way classification of antigen specificity (healthy donor, influenza, SARS-CoV-2), CurrAb is more than 10 percentage points more accurate than the next-best model on the paired task. The strategy comparison found that curriculum and constant-mix training both beat the fine-tuning approach at 650M parameters, consistent with the fine-tuned model losing unpaired-sequence competence and overfitting the smaller paired set.

#Applications

The model is intended for antibody discovery and repertoire analysis workflows where paired-chain sequencing data is available: classifying B-cell receptor sequences by antigen specificity, scoring whether a heavy/light combination is a plausible native pairing, and producing embeddings for downstream developability or clustering models. Because the checkpoint is a standard masked-LM encoder distributed on Hugging Face under an MIT license, it drops into existing fine-tuning pipelines without modification.

#Impact

CurrAb's contribution is as much procedural as it is a checkpoint. By training the curriculum, constant-mix, and fine-tuned variants under matched conditions and releasing all three, the work gives the antibody modeling community a controlled answer to a question that had been settled by convention rather than measurement. Code is MIT-licensed on GitHub, and the Zenodo deposit carries both the weights and the pretraining corpora, so the schedule sweep can be reproduced rather than taken on trust. Evaluation is confined to in-silico classification and masked-token tasks; the model predicts neither structure nor binding affinity, and no experimental validation of its representations accompanies the release.