Genomic Context BERT

A restriction endonuclease and an anti-phage abortive-infection protein can share almost nothing in sequence, yet both sit inside a defense island, flanked by the same recognisable cast of neighbours. Sequence-based function predictors cannot see that: they treat each protein as an independent unit and ask only what its amino acids resemble. In prokaryotes that discards real signal, because co-regulated genes are packed into operons and syntenic blocks survive long evolutionary stretches.

Genomic Context BERT takes that idea to its limit by removing sequence from the model entirely. Every protein in a large prokaryotic corpus is clustered at 30% identity, each cluster becomes a single vocabulary token, and a genome is rewritten as an ordered sentence of those tokens. A BERT encoder is then pretrained from scratch with masked language modelling: predict the hidden gene from the genes around it. A gene's embedding is therefore a function of its neighbourhood rather than its own sequence, so one protein cluster receives different representations in different genomic settings. Daulet Toibazar, Maxat Kulmanov and Robert Hoehndorf of the bio-ontology group at KAUST posted the work to bioRxiv in October 2024.

The context-only design distinguishes it from its neighbours in the literature. gLM tokenises genes by their ESM-2 embeddings, mixing sequence with context; the same lab's DeepGO-SE predicts Gene Ontology terms from sequence and ontology axioms. Holding sequence out lets the authors ask what those hybrids cannot: how much of a context model's accuracy is functional inference, and how much is memorised synteny between related genomes?

#Key Features

• Protein clusters as a vocabulary: 119.8M proteins from 31,002 NCBI genomes were clustered with MMseqs2 linclust at 30% identity; after dropping clusters under 20 members, 544,993 remain, one token each.
• Contextual, not static, gene representations: a case study on the Cas2 family (IPR010152) finds one embedding cluster far from the rest, consistent with a role beyond adaptive immunity.
• Phylogeny-stratified evaluation: whole families, orders and classes are held out of training, progressively removing the synteny an ordinary random split would leak.
• Strongest where function is contextual: defense and secretion system proteins are predicted better than general enzymes, the opposite of the ordering sequence embeddings produce.
• Extends to Gene Ontology: embeddings from proteins with experimental QuickGO annotations feed a classifier over all three GO branches.

#Technical Details

The encoder is a BERT-base variant with 12 hidden layers, a hidden size of 512 and 8 attention heads over a 544,998-token vocabulary, roughly 330M parameters, most of them in the embedding table. Pretraining used masked language modelling with a 20% masking rate on nine-gene sentences, split 90/10 for training and validation, across approximately 30,000 bacterial genomes on 8 V100 GPUs. Fill-mask evaluation over 500 randomly selected genomes reached 94% top-1 and 99% top-5 accuracy. For function prediction, the embedding of the centre gene in each sentence feeds a three-hidden-layer MLP trained with binary cross-entropy against InterPro labels.

Against ESM-2 embeddings from esm2_t30_150M_UR50D, context embeddings reach Fmax 0.659 on defense, 0.595 on secretion and 0.596 on general proteins, versus 0.136, 0.162 and 0.214 for sequence; AUC is 0.968/0.966/0.950 against 0.829/0.880/0.937. Those numbers fall sharply once synteny is removed: defense-protein Fmax drops to 0.356 at family level, 0.342 at order and 0.281 at class, while general proteins fall to 0.249. Sequence embeddings barely move across the same splits. A second checkpoint, pretrained on the genome corpus of Miller et al., beats contextualised word2vec on nine KEGG functional classes, F1 0.870 against 0.692.

#Applications

The natural use is annotating uncharacterized open reading frames in newly assembled bacterial genomes and metagenomes, where sequence homology is thin but gene order is intact — anti-phage defense systems and secretion machinery in particular, defined largely by the operons they occupy. Because the signal is orthogonal to sequence, the embeddings also complement sequence-based annotation: 17 of 20 Biological Process terms, 15 of 20 Molecular Function terms and 12 of 15 Cellular Component terms scored higher AUC here than under DeepGO-SE, including plasmid recombination (GO:0042150), a function determined by the genes flanking the recombination site.

#Impact

The most durable contribution here is diagnostic. By isolating context from sequence, the work shows that genomic-context models can score well by memorising syntenic arrangements seen in training, and it supplies the phylogeny-stratified protocol that measures how much of a result rests on that shortcut — a control the word2vec and hybrid context models preceding it did not apply. The caveats are real: coverage is prokaryotic only, the vocabulary is fixed to clusters derived from a January 2023 NCBI snapshot, so a new genome must first be mapped onto them, and accuracy on general enzymes degrades outside the training clades. The work remains a preprint, single-version and not peer reviewed, with training code, pretrained weights and the formatted genome corpus all released.