ProCALM

An enzyme engineer who wants a bacterial version of a reaction only ever annotated in eukaryotes has nothing to type into a prompt-conditioned protein language model. Methods like ZymCTRL steer generation with a control tag — an EC number prepended as a token — and a token the model never saw during training carries no meaning, nor does it sit anywhere in particular relative to the tags that were seen. ProCALM (Protein Conditionally Adapted Language Model) replaces that tag with a learned continuous vector: a small conditioning encoder maps a function representation into a latent space, and that latent is injected into every transformer block through a trained adapter. Similar functions land close together in that space, so the model can be asked for something it was never trained on and still generate plausibly.

ProCALM was developed at Profluent by Aadyot Bhatnagar, Jeffrey A. Ruffolo and Ali Madani, with first author Jason Yang of Caltech, who did the work while at Profluent. The preprint appeared on arXiv in October 2024 and was revised in June 2025. Rather than train a conditional model from scratch, ProCALM finetunes ProGen2-base with conditional adapters while the original weights stay frozen — the parameter-efficient strategy previously used to condition protein language models on structure, extended here to function.

The three modalities explored — EC number, taxonomy, and natural-language descriptions of function — run through the same architecture with only the conditioning encoder's input dimension changing, making conditioning a pluggable representation rather than a vocabulary decision baked into pretraining.

#Key Features

• Function conditioning beyond tokenized tags: The conditioning encoder accepts any fixed-length representation of function, so EC one-hots, reaction fingerprints, and contrastively learned text-sequence embeddings all drive the same generator.
• Generalization to rare and held-out conditions: Prompted with EC classes seen fewer than ten times, or held out entirely, ProCALM still produces valid enzymes whose predicted function sits close to the target in the EC hierarchy.
• Joint conditioning through parallel adapters: A variant passes taxonomy and EC number through separate adapters whose updates are summed, so the model can be asked for an EC class in a kingdom where it was never observed, at some cost to single-condition accuracy.
• Parameter-efficient training: Only the adapters and conditioning encoder are trained, roughly 70 million parameters on top of a frozen 764M-parameter backbone.
• Text-guided generation: Given the same training data and text-protein embedding as ProteinDT, ProCALM produced more valid, correctly conditioned sequences from natural-language prompts, especially for rarely described functions.

#Technical Details

The backbone is ProGen2-base, a decoder-only autoregressive transformer with 27 layers, embedding dimension 1536 and 764 million parameters. The conditioning encoder is a two-layer MLP of hidden dimension 256; its output is concatenated with a dimension-16 low-rank projection of the hidden state and passed through a three-layer adapter MLP inside each transformer block. Training is ordinary autoregressive residue prediction. Conditioning representations tested were a 630-dimensional one-hot of the four-level EC hierarchy, a four-dimensional kingdom one-hot, 2048-dimensional DRFP reaction fingerprints, and 256-dimensional CREEP multimodal embeddings.

Training data came from UniProt as of June 17, 2024: a Swissprot Train split of 152,763 sequences spanning 4,201 EC numbers, and a UniRef set of 29.4 million EC-annotated sequences covering 5,222 ECs. Training ran across four 40GB A100s at 144k tokens per batch — 1.5 billion tokens in about 40 A100-hours for the Swissprot model and 9 billion in about 240 for the UniRef one, against roughly 15,000 A100-hours for ZymCTRL trained from scratch. On common EC classes ProCALM matched ZymCTRL on the fraction of valid, correctly conditioned enzymes and on the pLDDT of ESMFold-predicted structures, while producing more diverse sequences. Perplexity on held-out clusters tracked training-set perplexity, indicating the adapters do not overfit. Scaling the backbone to ProGen2-large and ProGen2-xlarge lowered loss but did not measurably improve generation quality or diversity.

#Applications

The practical target is enzyme discovery for reactions with no well-populated family to finetune on: new-to-nature catalytic activities, functions annotated only by sequence, and enzymes wanted in a host organism where that chemistry has not been observed. Because conditioning accepts natural language, a campaign can be specified in a sentence rather than an EC number; because training is cheap, a group can retrain on its own conditioning representation without touching the pretrained backbone.

#Impact

ProCALM's contribution is architectural rather than a benchmark record: adapter-based conditioning, previously demonstrated for structure, transfers to function and buys out-of-distribution reach that prompting does not. The reach is partial: almost none of the sequences generated for entirely held-out EC classes map exactly to the target, only to nearby functions, and no wet-lab validation accompanies the preprint, which remains under review with no journal version. Code and adapter checkpoints for six conditioning variants are public, making it straightforward to try the strategy on other autoregressive protein language models such as ProtGPT2 or ProGen3.