PARA

Most of an antibody Fv sequence is framework. Mask 15% of it — the BERT default used by nearly every antibody language model — and most hidden residues land in conserved regions an immunologist could fill in by aligning against germline. The task is solved before the model has to learn anything about CDR-H3, the loop whose sequence decides what the antibody binds. PARA (Pre-trained model of Antibody sequences trained with a Rational Approach) treats the masking budget as a design variable rather than a borrowed constant: it masks 70% of each sequence and spends that budget on CDR-H3 first, span-masking the remainder up to the target rate.

PARA was developed at the XtalPi Innovation Center in Beijing by Xiangrui Gao, Changling Cao, Chenfeng He and Lipeng Lai, first posted to bioRxiv in January 2023 and published in Frontiers in Immunology in October 2024. It sits in the antibody-specific pretraining lineage alongside AntiBERTa, AntiBERTy and AbLang, and is benchmarked directly against the latter two. The deliverable is the encoder itself — a fixed checkpoint whose latent representations feed downstream models — not an end-to-end predictor.

A second departure is positional encoding. PARA is built on DeBERTa's disentangled attention, which scores content-to-content, content-to-position and position-to-content interactions separately over relative distances. Absolute encodings, the authors argue, let a model memorise residue distributions at fixed numbering positions — a shortcut that collapses whenever a sequence is trimmed or numbered unconventionally.

#Key Features

• PARA mask: On heavy chains, part of CDR-H3 is masked first and span masking fills the rest to the target rate; light chains, being more conserved, get untargeted span masking.
• Masking rate chosen by ablation: Rates of 15%, 30%, 50%, 70% and 90% were trained and compared. Recovery improved through 70% and degraded at 90%, where too little sequence remains to condition on.
• Robustness to truncation: With relative position encoding, top-1 CDR-H3 recovery is 48.7% intact and 48.3% after trimming the first 10 and last 5 residues; the absolute-encoding variant falls from 46.8% to 38.2% under the same trimming.
• Frozen-encoder downstream use: Heavy-light pairing is learned by an InfoNCE-trained projection head over averaged embeddings with the encoder frozen; antigen binding uses only the CDR-H3 slice.
• Works with very small labelled sets: Classifiers built on PARA embeddings beat a from-scratch baseline on HER2 binding with only 150 training antibodies, and the Emibetuzumab mutant task used a 100-sequence split.

#Technical Details

PARA is a DeBERTa-V2 encoder with 12 layers, hidden size 512, 8 attention heads, a 2048-unit feed-forward width, a 25-token vocabulary and a relative-distance clamp of k=128, over sequences up to 150 tokens; the released checkpoint is roughly 44.7 million parameters. Training data are human Fv sequences from the Observed Antibody Space, clustered with Linclust at 80% identity and singletons re-clustered at 50%, leaving approximately 18 million sequences — 13.5 million heavy and 4.5 million light — augmented by randomly truncating 0-3 residues from each terminus. Pretraining ran 200 epochs with AdamW at a 3e-4 base learning rate and batch size 784, over two weeks on four 32GB V100s. On heavy-light pairing, PARA representations gave higher auROC and auPRC than AntiBERTy and AbLang under an identical head and recipe; on the full HER2 set, however, all three encoders beat the logistic-regression baseline from Mason et al. by a margin the authors describe as not significant.

#Applications

The immediate uses are antibody engineering and repertoire analysis. Infilling masked CDR positions narrows the combinatorial space a library design has to cover, and the framework-recovery results mean the model stays usable on the truncated entries common in repertoire data. The frozen embeddings serve as features for developability, affinity and specificity classifiers where labelled data run to hundreds rather than thousands of measurements — the regime most therapeutic programmes operate in. The pairing model addresses a specific gap in bulk repertoire sequencing, where heavy and light chains are observed but their native partners are not.

#Impact

PARA's contribution is methodological rather than a scaling result: it shows that the masking rate and placement copied wholesale from natural language pretraining are poorly matched to a substrate as positionally redundant as an antibody framework, and that fixing the objective can matter more than adding parameters. Weights and an inference script are on GitHub. Limitations are real: a single 44.7M-parameter checkpoint with no size variants, heavy and light chains tokenised identically with no chain marker, evaluation confined to human antibodies and two binding datasets, and downstream gains over a logistic-regression baseline that only open up in the low-data regime. The repository carries no license file and the preprint was posted under "no reuse allowed without permission", so the article's CC BY 4.0 terms cover the text while nothing states the terms for the weights.