Genomic foundation model using the Hyena operator to process DNA at single-nucleotide resolution with context lengths up to 1 million tokens, 500x longer than transformer-based predecessors.
HyenaDNA is a genomic foundation model from HazyResearch at Stanford that fundamentally rethinks how neural networks process DNA sequences. Rather than relying on the transformer attention mechanism — which scales quadratically in memory and compute with sequence length — HyenaDNA is built on the Hyena operator, a sub-quadratic alternative based on implicit long convolutions. This architectural choice enables the model to process DNA at single-nucleotide resolution with context windows up to 1 million tokens, representing a 500x expansion over the 512–4k token limits of prior attention-based genomic models.
The central problem HyenaDNA addresses is one of resolution and scale. Transformer-based genomic models such as the Nucleotide Transformer have had to trade off one of two things: either they aggregate nucleotides into fixed k-mers (losing the resolution needed to capture single nucleotide polymorphisms and other fine-grained variants), or they accept severely truncated context windows (losing sensitivity to long-range regulatory interactions that can span hundreds of kilobases). HyenaDNA sidesteps this trade-off entirely by making long context computationally tractable at the single-nucleotide level.
The model was developed by Eric Nguyen, Michael Poli, and colleagues at HazyResearch and was presented as a Spotlight paper at NeurIPS 2023. It was pretrained on a single human reference genome (hg38) using a next-nucleotide prediction objective, and released with a family of checkpoints spanning context lengths from 1k to 1 million tokens. HuggingFace integration is provided through the LongSafari organization.
HyenaDNA's backbone is a stack of Hyena layers, each of which interleaves an implicit long convolution with pointwise nonlinearities and element-wise gates. The convolution filter is not stored explicitly; instead, it is generated on-the-fly by a small MLP that takes positional encodings as input and produces the filter weights. This implicit parameterization allows the convolution kernel to extend to the full sequence length without proportional memory growth. The resulting architecture achieves O(N log N) time complexity via the FFT-based convolution algorithm, and its memory footprint scales linearly with sequence length.
The model was pretrained on the GRCh38 human reference genome using a standard autoregressive next-token prediction loss, treating each nucleotide as a single token. Six checkpoint sizes are publicly available, paired with six context lengths: hyenadna-tiny-1k, hyenadna-small-32k, hyenadna-medium-160k, hyenadna-medium-450k, and hyenadna-large-1m-seqlen (6.6M parameters). The largest model was pretrained on a single node with 8 A100-80GB GPUs. Downstream evaluation follows a standard fine-tuning protocol with a classification or regression head appended to the pretrained backbone. On the GenomicBenchmarks suite HyenaDNA surpasses state-of-the-art on 7 of 8 datasets by an average of 9 accuracy points, and by up to 20 points on enhancer identification. On the 18-dataset Nucleotide Transformer benchmark, it reaches state-of-the-art on 12 datasets using a model with roughly 1,500x fewer parameters and 3,200x less pretraining data than the 2.5B-parameter Nucleotide Transformer.
HyenaDNA is designed for tasks that require either high resolution (single-nucleotide sensitivity) or long-range context (or both). Core applications include regulatory element prediction — identifying enhancers, promoters, and silencers from raw sequence — as well as chromatin accessibility profiling, transcription factor binding site prediction, and multi-label chromatin state annotation (e.g., the 919-way DeepSEA benchmark). The model's long context window makes it particularly well-suited for studying distal regulatory interactions, where an enhancer may sit hundreds of kilobases from the gene it controls. Species classification from raw genomic sequence is another demonstrated use case: by increasing context length to 1 million tokens, HyenaDNA can distinguish between species based on long stretches of genomic sequence that would exceed transformer limits entirely. The HuggingFace integration and Google Colab notebook lower the barrier for computational biologists to fine-tune the model on custom datasets without deep ML infrastructure.
HyenaDNA established that sub-quadratic sequence models — specifically those based on the Hyena operator — are viable and effective alternatives to transformers for genomic pretraining. Its demonstration of 500x longer context at competitive accuracy challenged the prevailing assumption that large transformer models with k-mer tokenization were the only path to genomic foundation models. The work directly influenced subsequent long-context genomic architectures and contributed to growing interest in state-space and convolution-based models in biology. A notable limitation is that pretraining on the human reference genome alone provides no cross-species generalization without additional fine-tuning, and the model does not natively model epigenomic or 3D chromatin structure. The model also does not incorporate multi-species or multi-omics data during pretraining, which later models such as Evo (Arc Institute) address with considerably larger compute budgets. Nevertheless, HyenaDNA's parameter efficiency, open-source availability under the BSD-3-Clause license, and clear demonstration of in-context learning in genomics make it a significant and widely referenced milestone in the biological foundation model landscape.
Nguyen, E. D., et al. (2023) HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution. Neural Information Processing Systems.
DOI: 10.48550/arXiv.2306.15794