GSM (Generation Sequence Model)

Ask a masked protein language model to write a sequence into an empty canvas and it will fill the blanks with whatever residues are probable in the abstract: nothing in the prompt says the result should be a serine hydrolase rather than a coiled coil. Natural-language captions are an awkward way to supply that missing conditioning: the same enzyme gets described a dozen ways, and a text encoder has to relearn what curators already recorded formally. GSM takes the prompt straight from the ontologies instead. An EC number, a set of GO terms, an InterPro family and a Gene3D domain each become a token in the Annotation Vocabulary, and the model's job is to emit amino acids consistent with that token set.

GSM was built by the Gleghorn Lab at the University of Delaware and described in the July 2024 Annotation Vocabulary preprint, alongside the contrastive representation model CAMP, which shares the vocabulary but curates an embedding space rather than generating. GSM is an encoder-decoder: a 12-block Annotation Transformer encodes the prompt, a modified ESM-2 150M is the decoder, cross-attending to the encoder's last hidden state while it denoises masked residues. Using a bidirectional BERT-style model as the decoder means generation proceeds by repeatedly unmasking, closer to a diffusion schedule than to left-to-right sampling.

#Key Features

• Ontology tokens as the prompt: Conditioning arrives as sorted integers drawn from EC, GO, InterPro and Gene3D, so the design specification is the controlled vocabulary a curator would write, with no natural-language paraphrase in between.
• Cross-attention instead of a merged vocabulary: The paper's alternative, ASM, extends ESM-2's embedding matrix with annotation tokens and pays self-attention cost quadratic in the combined length; cross-attention keeps the sequence track's length dominated by amino acids.
• Parallel unmasking: Each forward pass fills the top-u mask positions ranked by logit or entropy; ten at a time rather than one cuts the cubic cost of fully greedy denoising without materially hurting quality.
• An alignment score built for design: Generated and ground-truth sequences are compared by a Needleman-Wunsch alignment under BLOSUM62, rescaled to run from zero to one, so a domain recovered slightly out of frame is not punished the way exact-match accuracy punishes it. Randomly paired sequences average about 0.15.

#Technical Details

The Annotation Transformer was first trained for two epochs on 212,000 annotation-only entries, then joined to the decoder and trained further, both tracks together, on the FINAL corpus: 700,000 UniRef50 representative sequence-annotation pairs up to 512 residues, plus 157,000 experimentally validated pairs up to 512 and 104,000 between 512 and 2,048. Training ran in two stages: eight epochs at a maximum length of 512, learning rate 1e-4 and batch size 32, with the sequence mask rate drawn from a normal distribution centered at 0.5 and clipped to 0.15-1.0; then two epochs at a maximum length of 2,000 and learning rate 1e-5, with the annotation track unmasked so its language-modeling head enforced identity on the embeddings. ESM-2 was modified with extra layer norms on the self-attention queries and keys and SiLU activations for stability.

Evaluation fed 1,000 held-out test sequences at 15, 30, 50, 70 and 100 percent masking, with GSM additionally receiving the full annotation prompt. Below 50 percent masking ESM-2 150M scored higher on the alignment metric; at 70 percent and from complete noise GSM was substantially ahead of both ESM-2 and random mask filling. BLAST queries against nonredundant SwissProt returned statistically significant hits for many generations, and Blast2GO enrichment on those hits recovered GO terms matching the prompt even for sequences with low identity to the entire training set.

#Applications

The intended use is annotation-conditioned protein design: state the catalytic activity, the cellular compartment and the domain architecture you want, and read out candidate sequences, either from scratch or by masking a region of an existing scaffold and letting the prompt guide the fill. Because the prompt vocabulary is UniProt's own, a group specifies a target with labels it already assigns rather than devising a prompt format, and the same machinery does scaffold completion: hold a conserved core fixed and let variable regions regenerate under a functional constraint.

#Impact

GSM is a prototype and the authors present it as one. Generation is bimodal: outputs either recover recognizable conserved domains or collapse into self-reinforcing repetition, screened by a chi-square test against the training-set amino-acid distribution. No generated sequence was synthesized or assayed, the work remains a preprint that has not been peer reviewed, and the trained weights have not been released: the repository's training script says so directly, and the model class it imports is absent from the lab's public utility package. What carries forward is the demonstration that a bidirectional protein encoder can be driven generatively by a structured functional prompt — a thread the lab continues in DSM, where masked diffusion replaces the cross-attention scheme.