OmicsFUSION

OmicsFUSION is a pretrained genomic framework that encodes DNA sequence and functional omics signals in a single model. It was developed at the International Bioinformatics Laboratory of HSE University in Moscow with InsideOutBio, and posted to bioRxiv in September 2025.

The motivation is a structural limitation of sequence-only DNA language models. Because the genome is essentially identical across cell types, models such as DNABERT, Nucleotide Transformer, and Evo 2 cannot on their own resolve why the same stretch of DNA behaves differently in different tissues — that distinction is carried by chromatin accessibility, histone modifications, transcription factor occupancy, and other epigenomic layers. OmicsFUSION takes those tracks as a primary input alongside the sequence.

Architecturally the model follows HyenaDNA in replacing self-attention with Hyena blocks, reducing the cost of long genomic context from quadratic to O(L log L) time. On that backbone the authors extend masked language modeling to omics tracks, so the model reconstructs missing epigenomic signal as well as missing nucleotides, and add a separately trained contrastive module producing embeddings for arbitrary genomic intervals.

#Key Features

• Joint DNA and omics encoding: DNA 6-mer tokens and per-position omics feature vectors are projected into a shared embedding space and processed by one encoder, replacing manual integration of individual signal tracks.
• MLM extended to omics tracks: DNA masking follows BERT (15% of tokens, in contiguous spans of at least six overlapping 6-mers), while omics masking zeroes out up to 10% of entire feature tracks across a region, since neighbouring omics values are too correlated for token-level masking to be informative.
• Two embedding granularities: The encoder yields fused single-nucleotide embeddings; a frozen-encoder contrastive model trained on proximity- and function-based interval pairs yields interval-level embeddings via mean pooling.
• Released model family: Six encoder-decoder checkpoints span two genomes (hg38, mm10) and three embedding widths (384, 512, 768), from 28.5M to 99.1M parameters.
• Curated omics corpora: Human and mouse datasets aggregated from ChIP-Atlas are released alongside the models, pre-batched and chunked for multi-GPU and multi-node training.

#Technical Details

The encoder is a 12-layer stack of Hyena blocks with rotary positional embeddings on the DNA branch and a linear projection on the omics branch. The decoder shares three convolutional layers before splitting into ConvNeXt-based DNA and omics heads. Training combines cross-entropy on masked DNA tokens with binary cross-entropy on omics values, weighting masked omics targets three times more heavily than unmasked ones. A staged curriculum grew the context window from 512 to 8,192 to 65,536 base pairs over two epochs per stage, using AdamW at a peak learning rate of 1e-3 and BF16 mixed precision on H100 and A100 nodes for more than 50,000 GPU hours.

Training data came from ChIP-Atlas: over 60,000 human and over 40,000 mouse experiments, grouped by omics feature and tissue, aggregated by maximum signal, and filtered to groups with at least three experiments — yielding roughly 1,000 human and 500 mouse tracks, with ENCODE blacklist regions excluded. For the largest human model, non-masked DNA token accuracy reaches 0.915 and omics reconstruction ROC AUC 0.998, while masked-region metrics are lower (0.403 token accuracy, 0.560 omics F1). On four tasks from the Genomic Benchmarks suite, OmicsFUSION improves over the published baselines: 84.4% vs 68.9% accuracy on human enhancers (Cohn), 86.4% vs 81.1% on human enhancers (Ensembl), 88.6% vs 86.5% on non-TATA promoters, and 86.8% vs 68.8% on open chromatin regions.

#Applications

The framework targets researchers who need epigenomic context in a downstream model but do not want to engineer it track by track. A single fused embedding can feed classifiers for enhancers, promoters, and open chromatin regions; the decoder can impute omics signals for tissues or assays absent from the training set; and because the representation is tissue-conditioned, it supports cell-type-specific modelling that sequence-only encoders cannot express. Interval embeddings additionally support similarity search over functional genomic regions.

#Impact

OmicsFUSION is a preprint that has not yet been peer reviewed, and its evidence base is narrower than its framing: four human classification benchmarks plus internal reconstruction metrics, with no comparison against contemporary DNA foundation models on a shared suite such as GUE. Reuse also carries friction — the model requires omics tracks as input, so it is not a drop-in replacement for sequence-only encoders where such tracks are unavailable. Code is released under GPL-3.0, but only through an anonymized AAAI submission repository with no README or usage documentation, and checkpoints and datasets are distributed via a Google Drive folder rather than a durable host such as HuggingFace or Zenodo; no model card or data card accompanies the release. The underlying argument — that jointly modelling sequence and epigenomic signal yields more transferable genomic representations than scaling sequence-only pretraining — is substantive, and the released multi-genome, multi-width checkpoint family gives it a concrete testbed.