GeneMoE

The enhancers that set a gene's expression level can sit hundreds of kilobases from its promoter, which is why sequence-to-expression models use windows of half a megabase or more. DNA language models are built the other way: attention cost grows with the square of sequence length, so pretraining budgets go to depth and width instead and most genomic encoders stop at a few kilobases — short enough that the distal element doing the regulating is not in the window at all. GeneMoE buys the extra context from sparsity rather than compute. Replacing each feed-forward block with a mixture of experts routes each token to just two of eight experts, so capacity rises while the per-token FLOP count does not, and the freed budget goes into a curriculum that walks pretraining from 2 kb windows out to 256 kb.

GeneMoE is a bidirectional, encoder-only DNA foundation model from GenBio AI, described in three sizes — 235M, 470M and 1B total sparse parameters. GenBio first described the family in October 2025 under the name AIDO.DNA2, and named it GeneMoE in the August 2026 preprint that introduces GB.GeneUnet, where it serves as the architectural control: comparable size, identical data, so the differences between the two are attributable to architecture rather than corpus or scale.

It is pretrained on OpenGenome2, the 8.84-trillion-nucleotide multi-species corpus assembled for Evo 2, spanning eukaryotic and prokaryotic genomes, metagenomes, organelle DNA and annotated functional elements — a marked change from GenBio's earlier AIDO.DNA, a dense encoder fixed at 4 kb and trained on the 796-genome Nucleotide Transformer set, and from supervised regulatory models such as Enformer and Borzoi, which see only human and mouse.

#Key Features

• Sparse expert routing: Eight routed experts plus one shared expert replace the standard feed-forward layer. A sigmoid router with a learnable per-expert bias selects the top two routed experts per token, with a load-balancing loss and a router z-loss keeping utilization even.
• Staged context curriculum: Pretraining steps the window through 2 kb, 8 kb, 64 kb and 256 kb rather than starting long; the 235M model finishes at 256 kb and the 470M and 1B models at 64 kb.
• Multi-species self-supervision: A masked-token recovery objective over OpenGenome2 uses no assay labels, so representations reflect evolutionary constraint shared across clades rather than one species' annotations.
• Frozen-encoder transfer: Downstream regulatory prediction runs by linear probing — the encoder stays frozen and only a lightweight adapter is trained.
• Zero-shot variant scoring: Pathogenicity is read off as the L2 distance between reference and alternate embeddings over the 128 nucleotides flanking the site, with no labeled variant training.

#Technical Details

The backbone is an encoder-only transformer with pre-LayerNorm, rotary positional embeddings, RMSNorm with zero-centered gamma and SwiGLU activations, retaining full-length attention at every layer. On the Borzoi benchmark, the 470M variant — called GeneMoE-medium in the paper's Methods and GeneMoE-470M in its figures — is frozen and coupled to a Flashzoi adapter over 524 kb inputs, with Flashzoi's convolutional DNA stem replaced by a linear projection to 512 channels and sliding-window attention of ±8,192 tokens inside the encoder; adapter training uses 32 H100 GPUs at a global batch of 32. Under that protocol GeneMoE outperforms both fully supervised baselines, Borzoi and Flashzoi, on CAGE, histone ChIP-seq, TF ChIP-seq, DNase and RNA-seq coverage. On zero-shot ClinVar variant effect prediction over roughly 41,000 pathogenic and 151,000 benign SNVs, GenBio reports the family ahead of AIDO.DNA and of Evo 2 at matched 1B scale. Perplexity is lowest at each variant's own pretraining context and climbs beyond it, so the 524 kb setting sits outside what GeneMoE was trained for.

#Applications

GeneMoE is a general-purpose genomic encoder for regulatory work: predicting coverage tracks — expression, chromatin accessibility, transcription-factor and histone occupancy — from sequence alone, scoring non-coding variants without labeled data, and supplying embeddings for phylogenetic and functional-element analysis. Because downstream use needs only a frozen forward pass plus a small trained head, groups without pretraining-scale compute can adapt it to their own assays and species.

#Impact

GeneMoE's clearest result is that a self-supervised encoder trained on no expression data at all, then linear-probed, beats regulatory models trained end-to-end for exactly that task — evidence that cross-species pretraining transfers to functional genomics. Its second contribution is a negative one GenBio published against itself: in the same preprint, GB.GeneUnet's hierarchical U-Net reaches lower perplexity at every context length and runs 35 to 100 times faster, because full-length attention at every layer is what GeneMoE pays for its sparse capacity. That is why the paper treats the mixture-of-experts design as a baseline rather than a destination. GenBio has released neither weights nor training code, so GeneMoE is at present a published result rather than an artifact others can run.