FAbCon

The scaling behaviour that holds across general-purpose language models had not been observed for antibodies: immunoglobulin-specific models were reported to plateau, or even degrade, beyond roughly 110 million parameters, a size general protein language models had passed years earlier. FAbCon, from Alchemab Therapeutics, tests whether that ceiling is a property of antibody sequence space or of the training setups used to reach it. Scaling from a 144-million-parameter configuration to 2.4 billion, antigen-binding accuracy keeps improving with size across three independent datasets.

The second design choice is how a paired antibody is represented. Earlier antibody language models such as AntiBERTa encode heavy and light chains independently, leaving the relationship between the two chains — the thing that shapes the combining site — outside the model's receptive field. FAbCon prefixes each chain with a dedicated token and concatenates them, so unpaired and paired antibodies share one token stream and can share a mini-batch. Because the model is decoder-only and causal, it can emit a complete paired antibody from the heavy-chain prefix token alone.

The work was presented at the ICML 2024 Workshop on Accessible and Efficient Foundation Models for Biological Discovery. Three sizes — small (144M), medium (297M) and large (2.4B) — are distributed on Hugging Face under a modified Apache 2.0 licence limited to non-commercial research: pre-clinical and clinical work, and the synthesis of reagent, diagnostic or therapeutic antibodies, are excluded.

#Key Features

• Paired-chain generation from a single prompt: Chain-prefix tokens let one causal decoder emit both variable domains as a single sequence, rather than generating each chain separately and pairing them afterwards.
• Scale that pays off for antibodies: FAbCon-large reaches a mean AUPR of 0.883 across the HER2, SARS-CoV-2 and IL-6 binding tasks, ahead of the closest competitor ProGen2-OAS at 0.847, with accuracy rising across the three FAbCon sizes.
• Pretraining carries the signal, not the architecture: Randomly initialised controls of matched size score 0.765 and 0.719 mean AUPR against 0.883, isolating what the pretrained weights contribute.
• Zero-shot humanness oracle: Sequence perplexity correlates with OASis humanness scores at Pearson r = -0.938, so the pretrained model ranks candidates for germline-likeness without any labelled data.
• Frozen-backbone downstream use: Every binding evaluation freezes the decoder and trains only a linear classification head, so gains reflect the pretrained representation rather than task-specific refitting.

#Technical Details

FAbCon is a decoder-only transformer built on the Falcon architecture with multi-query attention, trained with a causal language modelling objective for 200,000 steps. The three configurations are 24 layers / 12 heads / 768 dimensions (144M), 28 / 16 / 1024 (297M) and 56 / 32 / 2048 (2.4B). The vocabulary holds 26 tokens — the 20 amino acids, four special tokens and the two chain-prefix markers — inside a 256-token context window.

Pretraining drew 1.47 billion unpaired sequences from the Observed Antibody Space snapshot of 23 February 2023 plus 43.4 million proprietary BCR sequences from 376 individuals, clustered with Linclust at 90% identity across the VH or VL domain. Paired data combined 1.5 million public sequences with 1.4 million in-house pairs, filtered at 99% redundancy to 2.5 million. The final corpus held 823.7 million sequences, of which 2.5 million are paired; FAbCon-small and FAbCon-medium were pretrained on a public-only subset, making them the reproducible variants. On HER2, FAbCon-large holds an AUPR of 0.775 with 5% of the training set (912 examples), against 0.856 on the full set.

#Applications

The intended workflow is a closed loop: FAbCon generates a library of paired antibody sequences, a binding-fine-tuned copy of the same model ranks them as an oracle, and measured binding data feeds back to sharpen that oracle. One thousand de novo paired sequences generated by FAbCon-large fall inside the acceptable windows of the Therapeutic Antibody Profiler and show OASis humanness distributions resembling a human BCR repertoire, with heavy-chain CDR length distributions closest to those of human PBMCs. Discovery teams can also use it as a sequence encoder for repertoire triage, or perplexity alone to filter candidates for germline-likeness before assay.

#Impact

FAbCon supplies direct evidence that antibody-specific language models continue to benefit from scale past the point where earlier work reported saturation, and that specialisation buys accuracy that scale alone does not: FAbCon-large exceeds ESM-2 at 15 billion parameters on all three binding tasks while carrying roughly six times fewer weights. The paper is candid about where the approach stops: perplexity is a poor discriminator of binders, so the model becomes a usable oracle only once a minimum volume of antigen-binding data is available for fine-tuning. Reuse is constrained too — no training code has been released, FAbCon-large's corpus includes proprietary sequences that cannot be reconstructed, weights are gated behind a contact form, and the licence rules out therapeutic development. The work remains a preprint and workshop paper.