GoBERT

GoBERT is a BERT-style encoder that treats Gene Ontology (GO) terms themselves as the vocabulary of a language, learning the relationships among biological functions rather than the relationship between a sequence and a function. Given the set of GO terms already assigned to a gene or gene product, it predicts additional terms that gene is likely to carry. It was developed at the University of Texas at Arlington with Johnson & Johnson Innovative Medicine and published at AAAI-25.

Most automated function annotation works from the molecule outward: methods such as DeepGO map an amino acid sequence, a structure, or a family assignment onto GO terms. That framing ties prediction to proteins, and to the availability of sequence or structural evidence. GoBERT inverts the input. It consumes existing annotations and reasons over the ontology's own structure, which means it applies to any entity that can be annotated — protein-coding genes, non-coding genes, RNA products — and does not stall when sequence or structure evidence is thin.

The Gene Ontology is a directed acyclic graph of roughly 47,000 function terms spanning molecular function, cellular component, and biological process, arranged from general roots to specific leaves. GoBERT exploits both the graph's explicit edges and the implicit co-occurrence patterns that appear when the same functions are repeatedly annotated together across millions of genes.

#Key Features

• Annotations as input, not sequence: The model takes a gene's known GO terms as a token sequence, so prediction requires no amino acid sequence, structure, or family membership.
• Two complementary pretraining objectives: A self-supervised neighborhood prediction task, framed as multi-label classification against the GO adjacency matrix, captures explicit ontology edges; a specified masking-and-recovery task uncovers implicit associations among functions that the graph does not encode.
• LLM-initialized term embeddings: Token embeddings are initialized from large language model encodings of each GO term's text description, injecting semantic similarity between functions that are textually related but graphically distant.
• Cross-category inference: Because all three GO sub-ontologies share one vocabulary, molecular function and biological process annotations can be used to infer cellular component terms, and vice versa.
• Reusable function embeddings: The released checkpoint loads through the standard transformers API and emits function-level embeddings for downstream annotation and analysis tasks.

#Technical Details

GoBERT is a 10-layer bidirectional transformer with a hidden size of 1,024, 16 attention heads, and a vocabulary of 47,734 GO-term tokens, totaling roughly 205 million parameters. The implicit-relation corpus was assembled from UniEntrezDB, which contributes 5.1 million genes carrying 139 million functional annotations under unified Entrez Gene identifiers; after deduplicating annotation inputs, 413,000 genes with 6.1 million annotations were retained and partitioned into train, validation, and test splits by K-means clustering on averaged function embeddings. The explicit neighborhood-prediction labels come from the GO DAG adjacency matrix, down-sampled at a rate of 0.001 to balance the task. Each model was pretrained for 20 epochs.

Evaluation uses the novel function prediction task the paper introduces: mask a known annotation and rank candidate terms at the correct ontology depth. GoBERT reaches 34.08% top-1 and 57.47% top-5 accuracy overall, and 53.91% top-1 and 76.15% top-5 accuracy when candidates are restricted to the target depth. The reported comparisons are ablations rather than external baselines — removing the masking strategy, the neighborhood prediction objective, or the LLM-derived semantic initialization each degrades every metric, with the masking strategy contributing the largest share.

#Applications

GoBERT is aimed at the annotation gap: the large fraction of genes with partial GO coverage that would otherwise require wet-lab characterization. Curators and comparative genomics groups can use it to propose candidate terms for sparsely annotated genes across species, and to fill in one GO sub-ontology from the other two. The paper works through two yeast case studies — inferring cellular component terms for TOF2, where the model's top prediction at the target depth surfaced a chromosome association supported by an independent study but absent from the manual annotation, and predicting more specific descendants of general terms for MGT1.

#Impact

GoBERT stakes out a distinct position in function prediction by making the ontology, rather than the molecule, the object of representation learning, and by extending the task beyond proteins to genes and gene products generally. Its practical reach is correspondingly narrower than a general-purpose sequence foundation model: it is trained for one task family, and a gene with no existing annotations gives it nothing to condition on. Code and weights are released under the MIT license, with the checkpoint on HuggingFace loadable in a few lines, and the UniEntrezDB annotation corpus is deposited on Zenodo under CC-BY-4.0, so the full pipeline is reproducible.