HybriDNA

DNA modeling imposes two demands that pull against each other. Regulatory logic plays out over tens or hundreds of kilobases — enhancers act at distance, chromatin domains span megabases — so a genomic model needs an enormous receptive field. At the same time, a single substituted base can abolish a transcription factor site, so that field cannot be bought by compressing sequence into k-mer tokens. Attention gives resolution but scales quadratically; state-space models give length but historically lag on precise recall.

HybriDNA, from Microsoft Research AI for Science with collaborators including the Mamba authors Tri Dao and Albert Gu, resolves this by interleaving both. It is a decoder-only DNA language model whose layers alternate between grouped-query attention and Mamba2 selective state-space blocks, processing sequences up to 131kb at single-nucleotide resolution. Because it is trained autoregressively rather than with masked language modeling, the same checkpoint that scores existing sequence can also write new sequence — a combination that DNABERT-2 and the Nucleotide Transformer do not offer, and that long-context predecessors such as HyenaDNA and Evo approach with different architectural trades.

Checkpoints span 300M, 3B, and 7B parameters, and performance improves consistently across that range, following the scaling behavior expected of language models.

#Key Features

• Hybrid attention and state-space stack: Grouped-query attention layers are interleaved with Mamba2 selective state-space layers, combining the recall of attention with the linear-time sequence handling of SSMs.
• 131kb context at base resolution: Sequences up to 131,074 bp are processed without tokenizing nucleotides into k-mers, so single-base effects remain visible to the model.
• Echo embedding fine-tuning: For discriminative tasks the sequence is presented twice, letting a causal model form representations that see the full context at every position — a workaround for the left-to-right blind spot that hampers autoregressive encoders.
• Generative regulatory design: Generative fine-tuning produces synthetic cis-regulatory elements with targeted properties, demonstrated on yeast promoters and cell-type-specific human enhancers.
• Consistent scaling: The 300M, 3B, and 7B variants trace the expected loss and downstream-performance curves, so capability tracks compute rather than plateauing.

#Technical Details

Pretraining uses next-token prediction over a large multi-species genome corpus curated from the Nucleotide Transformer dataset, deliberately chosen over masked language modeling so that generation comes for free. Training proceeds in stages: an initial phase at 8,192-token context for 500k steps, roughly 250 billion tokens and about 1.5 epochs, followed by context extension to the full 131kb window. On understanding tasks, HybriDNA reports state-of-the-art results across 33 datasets drawn from the BEND, GUE, and Long-Range Benchmark suites, covering transcription factor binding, epigenetic mark prediction, splicing, and variant classification. Generative evaluation follows the regLM protocol for synthetic cis-regulatory element design. Weights for all three sizes are published on Hugging Face under an Apache 2.0 tag, and the modelling code is distributed there too, loaded through trust_remote_code. The companion GitHub repository is documentation only — a README and a requirements file, with no licence file of its own — so the embedding-extraction, generation, and fine-tuning recipes it describes point back to the Hugging Face artifacts rather than shipping a pretraining stack.

#Applications

The long context suits questions that short-window genomic models cannot express: scoring distal regulatory variants against their target promoters, annotating features across an entire locus in one pass, and analyzing sequence contexts where the relevant signal sits tens of kilobases from the site of interest. The generative side targets synthetic biology — designing promoters and enhancers with specified activity for yeast expression systems or cell-type-restricted human constructs — where candidates can be ordered and assayed directly. Because checkpoints load through the standard transformers API with remote code enabled, embedding extraction for downstream classifiers requires little bespoke engineering.

#Impact

HybriDNA is a useful data point that hybrid attention–SSM designs, which proved themselves on natural language, transfer to genomics where the sequence-length pressure is more extreme. Its combination of long context, single-base resolution, and native generation in one model narrows the gap between DNA models that read and DNA models that write. The results remain a preprint, benchmark numbers have not been independently reproduced, and the released artifacts document inference and fine-tuning rather than reproducing pretraining — so the corpus curation and training runs cannot be replicated from what is public.