CB-pLM

Ask a conditional protein language model to make a sequence less hydrophobic and you append a tag to its input. Nothing in the training objective compels the model to read that tag: it can minimise its loss perfectly well by ignoring the conditioning and leaning on sequence context instead. The designer turns the knob, the output shifts, and there is no way to tell whether the shift came from the property they asked for.

CB-pLM answers that by moving the concepts inside the network. A concept bottleneck layer sits between the transformer encoder and the output head, and every neuron in it is pinned to one named, human-readable concept — hydrophobicity, isoelectric point, helix fraction, a Gene Ontology molecular function, a taxon. Because the model must route its prediction through that layer, the concepts are load-bearing rather than advisory, and setting a neuron's value at inference time is a genuine intervention on generation rather than a hint.

The model was built at Prescient Design, the machine-learning group inside Genentech, with collaborators at UC San Diego, Guide Labs, and NYU, and published at ICLR 2025. It ships as CB-LBSTER inside the same LOBSTER library as that team's earlier masked and causal protein models, in four pretrained sizes: 24M, 150M, 650M, and 3B parameters.

#Key Features

• Concept-indexed neurons: The [CLS] hidden state feeds a concept network predicting 718 concept values, each multiplied by a learned embedding to form the "known" representation, so a neuron's activation is the model's estimate of that property.
• Test-time intervention: To steer a property, a single integrated-gradients pass ranks which tokens most influence the target concept; the top 5% are masked and regenerated with the concept value clamped to the requested level.
• Linear decoder: Known and unknown embeddings are concatenated and passed through one linear layer, so each concept's contribution to each amino acid reads straight off the weight matrix — which residues carry aromaticity, which respond to charge at pH 6 versus pH 7.
• Debuggable by inspection: A deliberately miscalibrated aromaticity concept shows up as a row of near-zero decoder weights, flagging a concept the model cannot control without running a single intervention experiment.
• Orthogonality network: A parallel branch produces token embeddings pushed to be orthogonal to the concept embeddings, discouraging the model from smuggling concept information past the bottleneck.

#Technical Details

CB-pLM is a BERT-style encoder trained with three losses: masked language modelling, mean-squared error on the normalised concept vector, and the orthogonality penalty. Training data combines UniRef50 with SWISS-PROT after deduplication; the 718 concepts are 159 protein-cluster labels, 162 cellular components, 140 biological processes, 106 molecular functions, 101 taxa, 36 organisms, and 14 sequence-computable properties from Biopython's ProtParam. The concept layer is trained independently of the downstream network to limit concept leakage, and Gaussian noise on token embeddings keeps the attribution used for coordinate selection faithful. All scales use rotary position embeddings, a 512-residue maximum length, and 25% masking; the 3B configuration is 26 layers, 2560 embedding dimensions, and 40 attention heads in bf16 with DeepSpeed stage 1.

On 10,000 held-out antibodies from the Mason dataset, the 3B model reaches perplexity 2.50 against 3.01 for ESM-2 at the same scale. Across more than 80 single- and multi-property control experiments, the 24M model achieves 97.6% intervention accuracy versus 75.9% for an equivalently sized tag-conditioned baseline, with roughly 3× the change in concept magnitude; on simultaneous control of GRAVY and net charge at pH 7 it reaches 94.2% against 59.9% for the next-best conditional architecture.

#Applications

The immediate use is developability engineering on therapeutic proteins. In the paper's case study the model redesigns Siltuximab, an approved anti-IL-6 antibody with a large surface hydrophobic patch, to lower its GRAVY index within an edit distance of five — matching purpose-built guided-diffusion designers while keeping sequence naturalness intact, despite carrying 718 concepts rather than being trained for that one objective. The same interface supports multi-property optimisation over solubility-, viscosity- and aggregation-linked properties, and the concept layer doubles as an audit surface: an engineer can check what the model has learned about a property before trusting it to move that property.

#Impact

Scaling a concept bottleneck to 3B parameters while retaining generative language modelling shows that a fixed interpretable layer costs little in pretraining quality — a result that transfers to any domain with annotated concepts. The tradeoff is that training requires concept-annotated sequences, and the unknown embedding needed for performance can still retain concept information that the orthogonality loss only discourages. Code is Apache-2.0 within the LOBSTER repository, with notebooks and an MCP server for concept lookup and intervention; the four checkpoints sit in a personal Hugging Face namespace without model cards or a stated weights licence, and the reported evaluations are computational rather than experimental.