ASM (Annotation Sequence Model)

Mask fifty residues in the middle of a hydrolase and ask a protein language model to fill them back in, and it must infer everything it knows about that protein from the residues that remain. It cannot be told the protein is a hydrolase, because its input has no slot where that fact would go. Yet for most well-studied proteins the fact is already written down: UniProt records an EC number, Gene Ontology terms, InterPro families and Gene3D domains, curated by hand and machine-readable. ASM's premise is that this is a vocabulary problem rather than an architectural one, and that the fix is to give each ontology member a token and let it sit alongside the amino acids.

ASM — Annotation Sequence Model — was built by the Gleghorn Lab at the University of Delaware and released as a preprint in July 2024, alongside three sibling architectures that share the same Annotation Vocabulary. The construction is deliberately unremarkable: take a pretrained ESM-2 checkpoint, resize its token embedding matrix so the annotation integers fit alongside the 20 amino acids, append a protein's annotation tokens to its sequence, and keep training by masked language modeling over the union. Self-attention then mixes the two tracks with no cross-attention block, no projection head and no second tower — which is what distinguishes ASM from CAMP, the same paper's representation model, where a frozen sequence encoder and a frozen annotation encoder are contrasted against each other and never share an attention operation.

Because corruption is applied to both tracks, one checkpoint learns two directions at once: mask the annotations and ASM predicts function from sequence, mask the residues and it reconstructs sequence conditioned on function.

#Key Features

• One token stream, two vocabularies: Annotation integers occupy the extended region of the same embedding matrix as the residues, so self-attention mixes function and sequence with no new machinery.
• Bidirectional mask filling: A single checkpoint recovers masked annotation categories from the remaining annotations and masked residues from full annotation context.
• Sequence-only inference: The weights exercised when no annotations are supplied are exactly the size of the base model, so ASM drops into an existing embedding pipeline unchanged.
• Truncation that preserves whole proteins: When a pair exceeds the context window, annotations are shuffled and dropped first, and the sequence is cut only if more than 75% of the annotations would otherwise be lost.

#Technical Details

Both released checkpoints start from ESM-2 35M — 12 layers, hidden size 480, 20 attention heads, rotary position embeddings — and differ only in vocabulary and training corpus. ASM35_EXP carries a 33,358-token vocabulary and 50,023,343 parameters; ASM35_RED carries 38,983 tokens and 52,728,968. The extra parameters are the enlarged embedding matrix and language modeling head, not depth or width. Both were trained with a 15% masking rate. ASM35_EXP ran 23 epochs over EXP, 70,395 pairs drawn from UniProt entries with experimentally validated annotations, at a maximum length of 2,048. ASM35_RED first saw roughly a quarter epoch of RED_ALL — 4.25 million pairs at length 768 — then two epochs of the 516,184-pair deduplicated RED set at length 1,536.

On the paper's linear-probe protocol, frozen ASM embeddings averaged 3.3% (EXP) and 2.2% (RED) higher F1 than base ESM-2 35M across the in-distribution tasks. Sequence reconstruction is where the annotation context pays: at 50% masking ASM35_EXP scored 0.24 F1 against 0.23 for ESM-2 150M, at 70% masking 0.08 against 0.03, and at 5% masking 0.39 against 0.30 for its own base model. Cross-entropy loss followed the same curve, so the gain is not an exact-match artifact.

#Applications

The distinctive workflow is inpainting a sequence you already know something about: proposing substitutions around an active site, designing a mutagenesis panel for an enzyme whose EC number and domain architecture are known, or completing a partial sequence whose family assignment is settled. The reverse direction suits partly characterized proteins, where some annotation categories are filled and others blank. Run alone, the sequence track serves as a drop-in encoder for probe training on an unannotated proteome.

#Impact

The authors are candid about where the recipe falls short. On annotation mask filling the sequence-free Annotation Transformer beats ASM35 on five of six categories even when ASM is given the entire sequence, which they attribute to under-training or to a pretrained checkpoint being a poor starting point for a dual vocabulary. ASM also underperforms base ESM-2 35M on the CLEAN vector search benchmarks and falls well behind it on residue-wise thermostability. The work remains a preprint, and its evaluation is entirely in-silico. What it establishes is narrower and still useful: vocabulary extension is a cheap way to make an existing protein language model conditionable on curated function, and the reconstruction results argue for trying it on checkpoints larger than 35M.