AbGPT

Antibody language models are usually trained on antibodies and nothing else. The reasoning is sound — the complementarity-determining regions are hypervariable and unlike anything else in the proteome, so capacity spent on immunoglobulins is capacity spent where it counts. The cost is that such a model never sees the chemistry an immunoglobulin fold shares with every other protein: the beta-sandwich packing, the buried hydrophobic core, the surface residues that keep a domain soluble. AbGPT takes the other route. It begins from ProtGPT2, an autoregressive model pretrained across general protein space, and fine-tunes it on natural B-cell receptor (BCR) repertoires, so the generator that results carries broad protein priors into a domain where sequence variability is extreme.

AbGPT was released in September 2024 by Desmond Kuan and Amir Barati Farimani at Carnegie Mellon University. Generation is driven by prompting rather than by conditioning tags: supplying four starting residues characteristic of a heavy or a light chain steers the model to continue autoregressively into a full-length variable domain of that type. That is a different interface from IgLM, which conditions on explicit species and chain-type tokens and infills spans within an existing sequence, and from p-IgGen, which generates heavy and light chains as a pair. AbGPT produces unpaired chains, one at a time.

The paper's demonstration is a pipeline rather than a benchmark table: prompted sampling, length and perplexity screening, then in-silico developability assessment of what survives. Run end to end, it yielded a library of over 15,000 BCR sequences.

#Key Features

• General-protein pretraining, antibody fine-tuning: the backbone is a protein language model trained across the whole protein universe, adapted to BCR sequences by continued causal language modeling rather than trained from scratch on antibody data.
• Chain type chosen by prompt: four starting amino acids drawn from the most frequent heavy or light chain N-termini are enough to select which chain the model generates, with no conditioning tags or architectural changes.
• Built-in quality filter: sequences are screened on chain-specific length windows and discarded above a perplexity threshold of 13.0, so the pipeline emits a curated library rather than raw samples.
• Developability assessed before synthesis: the released library is scored for solubility with CamSol, aggregation propensity with AGGRESCAN, structural confidence with AlphaFold2 pLDDT, and human-likeness with BioPhi's OASis metric.
• Open weights and a command-line tool: the checkpoint is on HuggingFace and the generation code ships as a pip-installable package, both under the MIT license.

#Technical Details

AbGPT inherits ProtGPT2's GPT-2 decoder architecture — 36 layers, 1,280-dimensional embeddings, 20 attention heads, a 1,024-token context and a byte-pair vocabulary of 50,257 in which one token spans roughly four amino acids — for approximately 738 million parameters. Fine-tuning used the pretraining corpus assembled for AntiBERTa: 71.98 million unique human BCR sequences curated from 61 Observed Antibody Space studies, comprising 52.89M unpaired heavy chains and 19.09M unpaired light chains, with the original 8:1:1 split merged into a 9:1 train/test ratio. Generation used top-k sampling with a repetition penalty and minimum lengths of 28 tokens for heavy chains and 20 for light; surviving sequences were kept at 110–140 residues (heavy) and 100–120 (light). On the resulting library, AlphaFold2 pLDDT was high and tightly distributed for full-length chains but lower and broader for CDR-H3, the region where antibodies lack deep multiple sequence alignments. Heavy OASis identity centred near 0.8 at a 10% prevalence threshold, and ANARCI classified the generated sequences as human. t-SNE embeddings placed them in clusters adjacent to but separable from the training repertoire, with IGHV3 over-represented relative to other V-gene families.

#Applications

AbGPT is a library-generation tool for early antibody discovery: rather than proposing a binder against a named antigen, it produces large numbers of plausible, human-like variable domains that downstream selection can screen. That fits campaigns where the bottleneck is diversity of starting material — seeding display libraries, or supplying realistic sequences for training developability predictors. Because the filtering step already removes low-confidence and poorly folding candidates, the output is closer to a pre-triaged shortlist than to raw model samples.

#Impact

AbGPT is an arXiv preprint that has not been through peer review, and its evaluation is entirely computational — no generated sequence was expressed or assayed for binding, so the developability claims rest on predictors rather than experiments. The authors are explicit about two further limits: the prompt-and-filter framework constrains how much of the antibody space the library covers, and the IGHV3 bias leaves the repertoire less balanced than a natural one. Its contribution is the transfer-learning argument itself — that a general protein language model is a better starting point for antibody generation than antibody data alone — placed alongside antibody-only generators such as IgLM, p-IgGen and FAbCon. The MIT-licensed weights and pip package make that argument cheap for other groups to test.