Omni-DNA

Genomic foundation models have converged on a single recipe: pretrain a sequence encoder on unlabeled DNA, then attach a fresh classification head and fine-tune it separately for every downstream task. That works, but it scales badly — each new task means another copy of the model — and it locks the output into whatever shape the head was built for, usually a class label or a scalar. Omni-DNA revisits the decoder-only autoregressive design that large language models use, so that one set of weights can answer many questions and emit answers in more than one format.

Developed by researchers at Microsoft Research and Imperial College London, with collaborators at the Vector Institute and University Health Network, Omni-DNA is a family of six base models spanning 20M to 1B parameters. Pretraining is plain next-token prediction over DNA. The second stage is what distinguishes it: instead of one fine-tune per task, the tokenizer vocabulary is extended with task-specific tokens and the model is fine-tuned on all target tasks simultaneously. Because the outputs are just tokens, those tokens can encode a class label, an English sentence, or a discretized image.

Against DNABERT-2, the Nucleotide Transformer, HyenaDNA, and Caduceus, Omni-DNA reaches the best reported score on 18 of 26 tasks across the Nucleotide Transformer and Genomic Benchmarks suites.

#Key Features

• One backbone, many tasks: A single multi-task fine-tune covers 10 acetylation and methylation prediction tasks, and beats models trained on each task individually.
• Vocabulary expansion instead of task heads: New tasks and modalities are introduced by adding tokens to the tokenizer and initializing their embeddings, leaving the architecture untouched.
• DNA-to-text annotation: The DNA2Function task maps a sequence to a written description of the gene's function, producing annotations a biologist can read rather than a class index.
• DNA-to-image generation: The Needle-in-DNA task shows the same mechanism emitting image tokens, demonstrating that the output space is not restricted to language.
• A size for every budget: Six checkpoints from 20M to 1B parameters share one training recipe, so the small models are drop-in substitutes when compute is tight.

#Technical Details

Omni-DNA adapts the OLMo decoder-only transformer, borrowing design choices from the LLaMA and OLMo families: non-parametric layer normalization without bias on the 116M and 1B models, RMSNorm on the rest, and a byte-pair-encoding tokenizer with an initial 4,096-token vocabulary rather than fixed k-mers. ALiBi relative position embeddings were tried and dropped after they slowed convergence. Context length is 250 tokens, extendable during fine-tuning. The six base models — 20M, 60M, 116M, 300M, 700M and 1B parameters — range from 8 to 16 layers and hidden sizes of 256 to 2048, and each was trained on 300 billion nucleotides. The pretraining corpus is NCBI's multi-species genome collection, exact-deduplicated down to 30 billion unique nucleotides and then cycled for multiple epochs. Three fine-tuned checkpoints are released alongside the base models: Omni-DNA-Multitask, Omni-DNA-DNA2Function, and Omni-DNA-DNA2Image, all derived from the 1B backbone. Weights are on HuggingFace under the MIT license.

#Applications

The multi-task checkpoint is the practical draw for genomics groups that need several annotations from the same sequence — histone acetylation and methylation states, promoter and enhancer calls, splice site detection — without maintaining a separate fine-tuned model for each. The DNA2Function checkpoint suits exploratory annotation of uncharacterized sequences, where a natural-language description is a more useful starting point than a probability vector. The smaller checkpoints make the family usable for screening large candidate sets on modest hardware.

#Impact

Omni-DNA's contribution is less about raw benchmark gains than about the output interface. By showing that a genomic model can be fine-tuned once for many tasks and can emit free-form text or images, it opens genomics to the instruction-following patterns that have reshaped natural language processing. The limits are real: the 250-token context is short next to long-range genomic models, the cross-modal tasks are constructed by the authors rather than drawn from an established benchmark, and the paper is a preprint. Still, the full family of weights and the fine-tuned checkpoints are public alongside fine-tuning and inference code, which makes the approach straightforward for other groups to build on — though the pretraining loop itself is not released, the repository deferring to the ai2-olmo framework for it, so the base models cannot be reproduced from scratch.