JanusDNA

JanusDNA is a DNA foundation model built to resolve a training-objective dilemma that has divided genomic language models. Autoregressive pretraining — used by Evo and HyenaDNA — is compute-efficient because every token contributes to the loss, but it only ever gives a nucleotide left-to-right context. Masked language modeling, used by DNABERT-2 and the Nucleotide Transformer, supplies bidirectional context but wastes most of each forward pass, since only masked positions produce gradient. DNA is inherently bidirectional: bidirectional promoters drive transcription in both directions and account for roughly 11% of human gene expression, so a purely left-to-right view discards real regulatory signal.

The model's answer is Janus modeling, named for the two-faced Roman god. A sequence is encoded twice — once left-to-right and once right-to-left — by two independent stacks, and a final fusion attention layer with a custom mask combines the two directional representations so that each nucleotide is predicted from the complete surrounding context without ever seeing itself. Every position contributes to the loss, as in autoregressive training, while the prediction target is informed bidirectionally, as in masked modeling.

JanusDNA was developed at the Berlin Institute of Health at Charité – Universitätsmedizin Berlin with collaborators at Shenzhen Technology University, Carnegie Mellon University, the Max Planck Institute for Heart and Lung Research, Freie Universität Berlin, Heidelberg University Hospital, and Fudan University, and released as a preprint in May 2025.

#Key Features

• Bidirectional context at autoregressive cost: The Janus fusion layer lets every token in a sequence contribute to the training loss while still being predicted from both upstream and downstream context.
• Hybrid Mamba, attention, and MoE stack: Each directional encoder alternates Mamba state space blocks with feed-forward or mixture-of-experts blocks (16 experts, replacing half the feed-forward layers), capped by a four-head fusion attention layer.
• Megabase context on one GPU: The architecture processes up to 1 million base pairs at single-nucleotide resolution on a single 80 GB GPU, with pretraining performed at a 131,072-token context length.
• Parameter efficiency: On the Nucleotide Transformer tasks, JanusDNA matches or beats models carrying 250 times more activated parameters, using under 2M activated parameters per variant.
• Long-range variant effects: On DNALongBench eQTL prediction over 450 kb windows, it achieves the best AUROC on 8 of 9 tissues, ahead of both Caduceus and the 252M-parameter expert model Enformer.

#Technical Details

JanusDNA is pretrained on the human reference genome (HG38) alone, using single-nucleotide tokenization and a 131,072-token context, deliberately matching the data scale of prior work so that architectural gains can be isolated. Model variants are sized to roughly 2M activated parameters to compare fairly against Caduceus and HyenaDNA baselines; sparse mixture-of-experts routing raises total capacity well above that while keeping per-token compute fixed. On the Genomic Benchmarks suite of eight regulatory-element classification tasks, JanusDNA sets the best result on three, with the remainder close to the leader on an already-saturated benchmark. On the 18 Nucleotide Transformer datasets covering histone marks, regulatory annotation, and splice sites, it leads on 12. A later revision attaching two MLP layers after the fusion layer improved histone-mark performance further, reaching 0.835 MCC on H3 and 0.688 on H3K4me3. On DNALongBench eQTL, the ~7.7M-activated-parameter variant reaches AUROC 0.914 on tibial nerve and 0.852 on tibial artery, against 0.683 and 0.741 for Enformer.

#Applications

The model targets regulatory genomics tasks where the relevant signal spans far more sequence than a gene body: classifying promoters, enhancers, and open chromatin regions; predicting histone modification states and splice sites; and scoring whether a noncoding variant affects the expression of a gene hundreds of kilobases away. Its efficiency profile is aimed squarely at groups without large GPU clusters, since a megabase context fits on a single accelerator and the benchmark results were obtained at single-digit-millions of activated parameters.

#Impact

JanusDNA demonstrates that a carefully designed pretraining objective and hybrid architecture can substitute for raw scale in genomics, an argument that matters because genomic corpora are far smaller than natural-language ones. The code is released under Apache-2.0 and pretraining weights for the MLP variants are deposited on Harvard Dataverse. Two limitations are stated by the authors: training is restricted to the human reference genome, so cross-species and population-scale generalization is untested, and the comparisons are run at small activated-parameter budgets, leaving behavior at billion-parameter scale unexplored. The work remains a preprint and has not been peer reviewed.