GB.GeneUnet

Sequence-to-expression models are pulled two ways. The regulatory elements that set a gene's output can sit half a megabase away, so the input window has to be long; the output is a per-base coverage track, so the representation has to stay fine-grained. A plain transformer encoder satisfies the second by holding every nucleotide at full resolution in every layer, then pays quadratically for the first — which is why most DNA foundation models stop at a few kilobases, short enough that the distal enhancer is outside the window entirely.

GB.GeneUnet resolves the tension with an architecture borrowed from image segmentation. Seven downsampling stages pool the sequence from 1 bp per token to 128 bp per token, the long-range reasoning happens at that coarse bottleneck where global attention is affordable, and seven upsampling stages walk back to single-base output while skip connections re-inject the detail pooling threw away. The result is an 837M-parameter encoder that accepts 1,048,576 bp at single-base resolution — and, because attention runs on short sequences at depth, one that is far cheaper to run than a flat encoder of comparable size.

The model comes from GenBio AI, which released it in an August 2026 preprint alongside GeneMoE, the mixture-of-experts encoder it uses as an architectural control: same corpus, similar size, so the comparison isolates the U-Net design. Both are pretrained on OpenGenome2, the 8.8-trillion-nucleotide multi-species corpus assembled for Evo 2 — a different bet from the supervised regulatory models Enformer, Borzoi and AlphaGenome, which train on human and mouse alone and cannot draw on regulatory grammar conserved across the rest of the tree.

#Key Features

• Hierarchical sequence compression: Average pooling with stride 2 at each of seven stages, with channel width growing from 512 to 2,048 as resolution falls, keeps the parameter budget on the layers where tokens are few.
• Local attention in the arms, global at the bottleneck: Encoder and decoder blocks use sliding-window flash attention over eight neighbouring positions; only the eight bottleneck blocks attend across the whole compressed sequence.
• One checkpoint, two transfer modes: The encoder can be frozen and read by a light adapter or unfrozen and fine-tuned end-to-end; both benchmarks start from the same weights.
• Multi-species self-supervision: Masked-token recovery at a 15% masking rate over OpenGenome2, with no assay labels, in two 3-trillion-token stages — the first on conserved, gene-dense regions, the second on whole-genome sequence concatenated into megabase-scale windows.

#Technical Details

The network is a symmetric Transformer U-Net: a stem, seven downsampling blocks, eight bottleneck blocks, seven upsampling blocks and an output block, over a 128-token vocabulary covering DNA, RNA and special tokens. Transformer blocks use grouped-query attention, rotary embeddings and a SwiGLU feed-forward layer; bottleneck blocks swap in full attention and a GELU network. Pretraining ran 6 trillion tokens on eight H100 GPUs. On the Borzoi benchmark at 524 kb the frozen encoder is read by a Flashzoi adapter, and that linear probe beats both Borzoi and Flashzoi across all tracks. On the AlphaGenome benchmark at 1 Mb the backbone is unfrozen and trained in two phases — 25,000 steps at 524 kb, then 20,000 more at 1,048,576 bp with NTK-aware RoPE rescaling. Scored on 1,576 held-out intervals against AlphaGenome's own released labels, the two models are level: mean Pearson correlation across eight assay categories is 0.646 for GB.GeneUnet against 0.649, with every per-category gap under 0.02 in either direction. AlphaGenome retains a clear lead on splice-site classification, 0.707 AUPRC against 0.671. Against GeneMoE, GB.GeneUnet reaches lower perplexity at every context length from 8 kb to 524 kb and runs 100× faster at inference than the 1B variant.

#Applications

The eight output categories — CAGE, DNase, ATAC, RNA-seq, TF and histone ChIP-seq, PRO-cap and splice-site usage — are the standard readouts for interrogating non-coding sequence: which regions are accessible in a given biosample, where transcription initiates, how a promoter's output changes when the surrounding megabase changes. Fine-tuning at 1 Mb runs on a single eight-GPU node, putting megabase-context regulatory modelling inside an ordinary academic group's hardware budget.

#Impact

The claim that matters is about cost rather than accuracy: a multi-species pretrained encoder matches a purpose-built supervised model on regulatory tracks while needing a far lighter fine-tuning procedure, which makes the pretrain-then-adapt recipe competitive on a task supervised models have owned. Several limits are real. The model predicts one-dimensional tracks and not chromatin contact maps; the narrow sliding window leaves long-range propagation entirely to the bottleneck; and it is not optimised for variant effect prediction, which its masked-language and track-regression objectives do not reward directly. GenBio AI describes GB.GeneUnet as under commercial development and has not released the weights, and no training or inference code accompanies the preprint, so the results stand on the paper alone.