Loading…
Pretrains on EC, GO, InterPro and Gene3D ontology tokens with no amino acids, building a protein feature space from curated function alone.
No providers recorded yet. Browse all providers
A curator closing out a UniProt entry writes down an EC number, Gene Ontology terms for molecular function, biological process and cellular component, an InterPro family and a Gene3D domain. That record is precise, machine-readable and the product of real experimental work — and a protein language model cannot read it, because its only input slot takes amino acids. AT inverts the arrangement: it discards the sequence, maps every member of those ontologies to a unique integer, and trains a transformer to recover annotation tokens that have been masked out. The resulting feature space is built on what proteins do rather than what they are made of.
AT — the Annotation Transformer — was built by the Gleghorn Lab at the University of Delaware and released in a July 2024 preprint introducing the Annotation Vocabulary and three architectures that consume it. AT is the base of that family: CAMP freezes an AT as one of two towers and contrasts it against a frozen ESM-2, GSM uses an AT as the prompt encoder its decoder cross-attends to, and ASM instead merges annotation tokens into ESM-2's own vocabulary. AT alone reads no residues.
Annotations are fed sorted by integer value, giving the token stream a fixed grammar over which masked language modeling asks a well-posed question: given an enzyme's GO terms and domain assignments, which EC number is missing?
Both released checkpoints are a single BERT-like transformer block with hidden size 384, intermediate dimension 2,048 and 8 attention heads. AT_EXP carries a 33,328-token vocabulary, learned absolute position embeddings and 15,542,832 parameters; AT_RED carries 38,953 tokens, 17,314,890 parameters, and rotary position embeddings chosen for the larger vocabulary. Both trained at a 15% masking rate — 100 epochs on EXP, 10 on RED, with early stopping on validation MLM accuracy at a patience of three. EXP is 70,395 UniProt entries with experimentally validated annotations; RED is 516,184 UniRef90 representatives with nonredundant ones. AT sees only the annotation half of each pair.
On exact recovery of 15% randomly masked tokens from held-out validation sets, AT_RED reached 0.774 accuracy and AT_EXP 0.690. Under the harder aspect-masking protocol, averaged over EC, MF, BP, CC, Pfam and Gene3D, AT_RED scored 0.491 F1 and AT_EXP 0.472, against 0.450 for ASM35_RED even when ASM was additionally handed the full amino acid sequence. AT_EXP was strongest where the label space is most constrained — 0.716 F1 on EC, 0.672 on Gene3D, 0.552 on Pfam — while AT_RED led on the Gene Ontology aspects. A third variant, the twelve-block AT_FINAL trained on 212,000 annotation-only entries, serves as GSM's encoder and reached 0.705 recovery accuracy; it is not released.
The direct use is annotation completion for partly characterized proteins. A protein whose domain architecture and localization are settled but whose catalytic activity is not can have its EC number proposed from the terms already assigned, and the same works in every direction across the six aspects. Inference costs one transformer block over a few dozen tokens, so sweeping a proteome's partial UniProt records is cheap. The second use is compositional: any architecture needing a function-space encoder loads an AT checkpoint frozen, as CAMP and GSM do.
AT's result is the one the authors flag as surprising: a model that has never seen a residue annotates better than one that has, beating ASM35 on five of six aspects even when ASM is given the entire sequence. They attribute the gap to ASM being under-trained, or to a pretrained ESM-2 checkpoint being a poor start for a dual vocabulary. The scope limit is inherent — AT presumes a protein that already carries curated UniProt terms, so it does not stand alone for an uncharacterized sequence the way its sequence-only siblings do. The work remains a preprint that has not been peer reviewed, its evaluation is entirely in-silico, and the authors note that the headline 0.716 EC figure is a plain F1 rather than the F1max probe benchmarks report. What carries forward is the vocabulary itself, which returns as a conditioning signal in the lab's later DSM diffusion model.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.