GENERanno

GENERanno is a genomic foundation model built specifically for gene annotation, developed by GenerTeam at Alibaba Cloud with collaborators at Zhejiang University and the Institute of Dataspace. Base checkpoints were released in February 2025 and the accompanying preprint was posted to bioRxiv in June 2025.

Metagenomics sequences DNA directly from mixed microbial communities, leaving annotation pipelines with short, fragmented reads drawn from many species at once. The established tools for locating protein-coding regions — GLIMMER3, GeneMarkS2, and Prodigal — are hidden Markov models tuned for assembled single-species genomes, and their accuracy drops on fragmented, multi-species input. Language-model alternatives such as GeneLM have struggled to consistently beat those HMMs, in part because their k-mer tokenizers obscure the exact base positions that gene boundaries depend on.

GENERanno targets both problems. It is a bidirectional transformer encoder trained with masked language modeling — architecturally the mirror image of GENERator-v2, the autoregressive genomic language model from the same group — that tokenizes DNA one nucleotide at a time, so every base has its own representation, and reads context in both directions, which calling a coding region inherently requires.

#Key Features

• Single-nucleotide resolution over 8 kb: Each of A, T, C, and G is its own token, preserving the base-level detail that boundary detection needs while still supporting an 8,192 bp context window with global bidirectional attention.
• Dual-strand coding-sequence annotation: Two parallel token-classification heads predict coding regions on the positive and negative strands from a single input strand, allowing overlapping genes on opposite strands to be resolved.
• Reference-free pseudogene identification: Distinguishes pseudogenes from active coding sequences directly from raw sequence, replacing comparative-genomics workflows that require curated reference databases.
• Assembly-free taxonomic classification: Assigns taxonomy from arbitrary genome fragments as well as from marker genes, bypassing the contig assembly and 16S rRNA identification steps of conventional metagenomic pipelines.
• Functional sequence training: Pretraining uses RefSeq-annotated functional regions rather than whole genomes, concentrating signal and avoiding the noise of non-functional DNA.

#Technical Details

GENERanno adapts a Llama-style transformer to bidirectional attention and masked language modeling: 26 layers, hidden size 1,280, intermediate size 3,520, 16 attention heads with 4 key-value heads, RoPE positional encoding, SiLU activations, and a 64-token vocabulary, totaling roughly 500M parameters. Unlike ModernBERT, which mixes local and global attention at the same 8,192-token context, GENERanno uses global attention in every layer. Pretraining covered 715 billion base pairs of prokaryotic functional sequence from RefSeq across two epochs (1.4 trillion tokens), with Flash Attention and ZeRO for efficiency; a separate eukaryotic base checkpoint was trained on 386 billion base pairs.

On bacterial chromosome annotation, the fine-tuned CDS annotator reaches 0.888 exact match rate and 0.890 boundary accuracy, against 0.841 and 0.848 for Prodigal and 0.806 and 0.816 for GeneLM. Applied zero-shot to 42 archaeal sequences from 31 species, it matches the in-sample accuracy of HMM methods and far exceeds GeneLM (0.764 versus 0.553 exact match). On the accompanying Prokaryotic Gener Tasks suite — gene fitness, antibiotic resistance, gene classification, and taxonomic classification — the 500M model leads DNABERT-2, Nucleotide Transformer v2, and GenomeOcean on most of the 18 tasks, including GenomeOcean-4B at eight times its parameter count.

#Applications

The primary use is annotating prokaryotic genomes and metagenomes: calling coding regions in assembled genomes, plasmids, and fragmented community sequence, then flagging pseudogenes and assigning taxonomy in the same pass. That combination suits microbiome surveys, metagenome-assembled genome curation, antibiotic resistance surveillance, and annotation of poorly characterized or newly sequenced organisms where reference databases offer little help. The base checkpoints also serve as general embedding extractors and fine-tuning starting points for prokaryotic sequence classification and regression tasks.

#Impact

GENERanno is a demonstration that a carefully scoped encoder can outperform both decades-old HMM annotators and much larger genomic language models on a task where nucleotide-level precision matters more than scale. The release is fully open: MIT-licensed code and four HuggingFace checkpoints, covering prokaryotic and eukaryotic base models plus fine-tuned CDS annotators for each, along with the benchmark suite. Two limitations are documented by the authors: the dual-strand labeling scheme cannot separate genes that overlap on the same strand, and the 33-species annotation evaluation reflects in-sample performance rather than a held-out test set. The preprint has not yet been peer reviewed.