PrefixProt

Text-conditioned generation is straightforward in natural language, where a prompt like "write a summary" is itself made of the model's own tokens. Protein language models have no equivalent handle: the vocabulary is 20 amino acids, so there is no spare symbol space in which to write an instruction. The usual workarounds are to fine-tune a separate model per property, which is expensive and fails when a property has only a few hundred known examples, or to bolt a natural-language interface onto a protein model, which introduces a second modality and its own failure modes.

PrefixProt, from the Harbin Institute of Technology (Shenzhen), takes the third route: it learns the control tag rather than writing it. A short sequence of continuous "virtual tokens" is prepended to the frozen protein language model's key-value cache and trained on examples of a single desired property, so the tag lives in embedding space instead of the amino acid alphabet. Because each tag is a small, independent tensor, tags trained separately can be concatenated or averaged at generation time to request several properties at once — an alpha-helical fold and antimicrobial activity, say — without ever training on proteins that have both. The work was first posted as a preprint in December 2023 and published in the Journal of Chemical Information and Modeling.

#Key Features

• Learned control tags: Prefix virtual tokens are optimized from property-labelled sequences, avoiding the need for a hand-written text instruction or a curated protein-text corpus.
• Frozen backbone: The underlying protein language model is never updated; trainable parameters amount to 1.18% of the model for a 100-token structural prefix and 0.24% for a 20-token functional prefix.
• Compositional multi-objective control: Separately trained prefixes can be combined by concatenation or averaging to generate proteins satisfying two constraints simultaneously, including combinations for which almost no natural training data exists.
• Effective in the low-data regime: On subsampled antimicrobial peptide sets of 50 to 2,000 sequences, prefix tuning holds up where full fine-tuning and text-guided models degrade.
• Structural and functional validation pipeline: The released code folds generated sequences with ESMFold, assigns secondary structure with DSSP, and relaxes and scores structures with PyRosetta.

#Technical Details

The base model is ProtGPT2, a GPT-2-large-style decoder with 36 transformer layers, 1,280-dimensional embeddings, and 12 attention heads per layer. Structural tags were trained on sequences drawn from the SCOPe database — 9,168 alpha-helix-rich, 14,531 beta-sheet, and 14,223 alpha-beta proteins — while functional tags used 4,505 experimentally validated antimicrobial peptides from DBAASP and 861 anticancer peptides from AntiCP2.0, plus 353 dual-activity peptides reserved for evaluating tag combination.

For alpha-helical generation, PrefixProt reaches a median helix content of 60.8% (versus 61.0% for LoRA) with the best Rosetta energy of the methods compared at -1.208 REU per residue, and a median pLDDT of 61.93, behind full fine-tuning at 65.70. On peptide function, it achieves the highest median predicted activity of any method tested — 0.90 for antimicrobial peptides scored by CAMP and 0.89 for anticancer peptides scored by AntiCP 2.0 — ahead of LoRA (0.89 and 0.88), the specialized generators PepLSTM (0.80), PepCVAE (0.74), and HydrAMP (0.67), and well ahead of the text-guided models ProLLaMA and InstructProtein, which collapse on the data-scarce anticancer task at 0.045 and 0.09. Median pLDDT for the generated peptides is 77.59 (antimicrobial) and 80.38 (anticancer). Concatenated alpha-helix and beta-sheet prefixes yield hybrid folds at a median pLDDT of 55.20, with more than half of the high-confidence structures exceeding 15% content of both secondary-structure types.

#Applications

PrefixProt is aimed at design campaigns where the target property is well defined but poorly populated: antimicrobial and anticancer peptide discovery, generation of fold-constrained scaffolds, and the harder case of requesting two properties at once. Practically, a group can train a prefix on a few hundred in-house labelled sequences in place of fine-tuning a full protein language model, then reuse and recombine that prefix across projects. The released code covers both prefix training and generation from released prefixes.

#Impact

The result of practical interest is that a small continuous prefix matches or beats LoRA and full fine-tuning on these tasks while touching a fraction of a percent of the weights, and that composing independently trained prefixes works at all — a cheap route to multi-objective control that does not require multi-property training data. The evaluation is entirely computational: activity is scored by predictors rather than assays, and structural quality by predicted confidence and Rosetta energy, so none of the generated peptides have been experimentally tested. The paper also notes that useful prefix length scales with dataset size, so the hyperparameter has to be retuned per property. The repository is small and carries no license file, and the trained prefixes are distributed through Google Drive links rather than a versioned model host.