DNA-MFM

Local chromatin accessibility is a sparse signal: across a 114 kb genomic window an ATAC-seq track sits near zero almost everywhere and spikes only where the chromatin is actually open. Hand it to a masked-reconstruction objective alongside a dense DNA representation and the model drives most of its loss down by predicting near-zero everywhere, never learning peak structure. The sparse modality collapses, and what looked like multimodal pretraining is sequence modelling with an inert second channel attached.

DNA-MFM is a self-supervised encoder built around that failure mode. Each genomic window becomes three co-registered token streams — DNA sequence embeddings, local chromatin accessibility (LCL) and a broader global accessibility track (GBL) — projected into a shared 256-dimensional space, and one Transformer encoder reconstructs masked positions in any stream from the unmasked positions in all of them. The fix for collapse is normalization: LCL values are scaled with global statistics estimated across the training windows rather than per window, preserving the relative magnitude of peaks and removing the incentive to predict a flat near-zero signal. The encoder is trained once and then frozen; every downstream result comes from a linear probe or ridge regression on the fixed window embedding it emits.

The model was developed by Yuyan Chen as sole author, across affiliations at Houston Methodist Research Institute, Weill Cornell Medicine and ModelsLive, and posted to bioRxiv in August 2026. It is a preprint and has not been peer reviewed.

#Key Features

• Global normalization for sparse modalities: Training-set-wide rather than per-window scaling lifts local accessibility reconstruction from a cosine similarity near 0.25 to 0.717, where masking-ratio tweaks, peak-weighted losses and an auxiliary peak-classification head all failed.
• A single shared encoder over three streams: DNA, LCL and GBL tokens carry positional and modality-type embeddings alongside a learnable CLS token, so self-attention crosses modalities inside one window rather than through a late fusion head.
• Reusable frozen embeddings: One 256-dimensional vector per genomic window, mean-pooled over unmasked tokens and reused unchanged across seven evaluations rather than refit per task.
• Modality-weighted reconstruction: Huber losses at masked positions only, weighted per modality to balance a 1,536-dimensional DNA stream against lower-dimensional accessibility patch tokens.

#Technical Details

All inputs derive from the EnformerCelltyping framework: genome-wide hg38 windows of roughly 114 kb. The DNA stream is not trained here — a pre-trained chopped Enformer supplies 1,536 dimensions across 896 positions, projected to 256. LCL is a 128-bp-resolution ATAC-seq track patched at size 8, GBL a broader track patched at size 16. Coverage-aware filtering leaves roughly 20,000 pretraining windows on chromosomes 1–19 and about 940 held-out windows on chromosomes 20–22 — a chromosome-level split of one dataset, not an independent cohort. The encoder is six pre-norm layers, hidden dimension 256, eight heads, feed-forward dimension 1,024, GELU and dropout 0.1, trained for ten epochs on one A100 40 GB with masking ratios of 0.75 / 0.30 / 0.75 for DNA / LCL / GBL, AdamW, warmup, cosine decay and a masking curriculum.

Masked reconstruction reaches mean cosine similarities of 0.963 for DNA and 0.972 for GBL, with LCL the bottleneck at 0.717. On the held-out chromosomes, peak detection AUPRC rises from 0.1547 for mean-pooled DNA embeddings and 0.1644 for a DNA-only encoder to 0.7095 — the 4.6-fold gain reported as the headline result — and regulatory activity classification F1 reaches 0.9835 against 0.6182 for DNABERT-2, 0.7358 for Nucleotide Transformer and 0.6606 for pooled Enformer tracks. Activity ranking is the exception: pooled Enformer tracks lead on Spearman correlation, 0.8606 against 0.8281. Externally, ClinVar AUROC is 0.6829 against 0.6141 for Enformer tracks; matched-subset Pearson correlations are 0.525 for GTEx eQTL and 0.3539 for PBMC caQTL effect sizes.

#Applications

It serves as a feature extractor for regulatory genomics in cell types with matched ATAC-seq: embed a window once, then attach a cheap linear model for histone-mark activity classification or ranking, peak detection and peak localization. Because a variant inherits the embedding of the window containing it, the same vectors serve non-coding variant prioritization and effect-size regression, carrying pathogenicity-related information that sequence-derived functional tracks alone do not.

#Impact

DNA-MFM's contribution is diagnostic more than competitive. It names a concrete failure mode in genomic multimodal pretraining — sparse, peak-shaped regulatory signals collapsing toward zero when aligned with dense sequence representations — and shows that a normalization choice, not extra capacity or a peak-aware loss, is what recovers them, a result that transfers to any model fusing accessibility, methylation or expression tracks with sequence embeddings. The caveats are real: the scale is modest by genomic foundation model standards, the sequence half is a frozen dependency on Enformer that inherits its resolution and cell-type coverage, local accessibility reconstruction remains partial, and the internal benchmarks split a single dataset by chromosome. Training and evaluation code is MIT-licensed, but trained checkpoints and the derived embedding library are not released.