CloneLM

CloneLM is an autoregressive antibody language model from New York University and BigHat Biosciences, released alongside the Clone-informed Bayesian Optimization (CloneBO) procedure it powers. It was a spotlight at the NeurIPS 2024 AIDrugX workshop.

Antibody engineering campaigns mutate a weak starting binder over successive wet-lab rounds, and the space of plausible mutations is combinatorially large while the measurement budget is small. Most sequence models used to guide that search are trained on repertoires of individual antibodies, so they can say whether a proposed sequence looks typical but not whether a mutation is the kind that improves a binder. CloneLM changes the unit of training data: instead of single sequences it models clonal families — the sets of related sequences the human immune system produces as it iteratively mutates and selects an antibody against a target. A model of how those families evolve is, implicitly, a prior over which mutations raise fitness.

Concretely, CloneLM takes a clonal family as one long token sequence of concatenated antibody sequences separated by a special token, and is trained to continue it. Given a starting candidate, it can therefore hallucinate the family that candidate might have belonged to, and the log-likelihood of the next member approximates that family's fitness function. This connects it to antibody language models such as IgLM, which the paper uses as an independent measure of how human-like a designed sequence is.

#Key Features

• Clonal families as the training unit: Each training example is an entire evolving family rather than one antibody, so the model learns what a maturation trajectory looks like, not just what a typical antibody looks like.
• Implicit fitness prior: Sampling a family that contains the starting candidate and reading off the model's log-probability yields a fitness function, following a martingale-posterior construction that avoids an explicit latent variable.
• Separate heavy and light chain models: Two checkpoints are published, so both halves of a variable domain can be optimized with a matched model.
• Frozen backbone inside CloneBO: The optimization loop conditions on prior measurements with a twisted sequential Monte Carlo procedure that biases generation letter by letter. Weights are never updated for a target; steering happens entirely at sampling time.
• Standard checkpoint interface: The published weights load through AutoModelForCausalLM, so the model can be used for generation or scoring outside the Bayesian optimization loop.

#Technical Details

Training data comes from the Observed Antibody Space database. Clonal families were annotated with FastBCR and any family with fewer than 25 sequences discarded, leaving 908 thousand heavy chain families and 34 thousand light chain families. Both models are 377-million-parameter autoregressive transformers following the Mistral architecture, with amino acids space-separated so that each residue is its own token. Test perplexity is 1.276 on heavy chain data and 1.267 on light chain data. In the in-silico benchmarks, CloneBO running over CloneLM optimizes binding and stability oracles significantly better than the next best method over 100 design steps (Mann-Whitney p = 0.018 for binding, p = 0.006 for stability), against ten baselines including LaMBO, Sapiens-guided greedy search and CMA-ES. In a wet-lab round, 200 sequences were designed from 1,000 prior measurements of binding and melting temperature; CloneBO designs were significantly more likely to be synthesizable, its strongest binder was beaten by only 2 of 997 previously measured sequences, and its most stable design exceeded every prior measurement.

#Applications

The intended use is an antibody lead-optimization campaign, where a starting binder has been found by display or immunization and successive rounds must improve affinity and thermostability on a fixed budget of assays. CloneLM supplies the prior; CloneBO folds in whatever measurements exist so far and proposes the next batch. Because the model is generative over sequences alone, it applies to targets with no structure and to properties, like stability in the human body, that structure-based design methods do not directly address. The heavy chain checkpoint can also be used on its own to score or sample humanized variants.

#Impact

CloneLM reframes antibody sequence modeling around the evolutionary process rather than the endpoint, and it is one of the few antibody optimization methods validated in vitro rather than only against in-silico oracles. Its limitations are stated plainly by the authors: the relationship between a clonal family's fitness and a laboratory measurement is assumed to be a simple affine one, conditioning on many measurements is computationally expensive so only the most informative are used, and a candidate is scored only by how likely it is to belong to some clonal family, which excludes information from measured sequences that could not plausibly share one. The published work is a preprint and a workshop paper rather than a peer-reviewed journal article.