JEPA-DNA

Most genomic foundation models are trained with token-level objectives—masked or autoregressive prediction of nucleotides—which force the model to recover exact sequence content. A model can become very good at that and still learn little about what a region does: reproducing the literal bases of a masked promoter rewards local token statistics, not the functional identity of the span. JEPA-DNA, from NVIDIA, brings the Joint-Embedding Predictive Architecture (JEPA) idea from representation learning into genomics. Instead of reconstructing masked nucleotides, a lightweight predictor maps the visible context onto the representation that a slowly-updated copy of the encoder assigns to the masked segment, shifting the learning signal from token recovery toward semantic alignment.

Rather than a single new architecture trained from scratch, JEPA-DNA is a model-agnostic continual-pretraining framework: it layers the joint-embedding objective on top of a backbone's own generative loss, so a DNA model a group already runs can be grounded further instead of replaced. NVIDIA released the pretraining and benchmarking code under Apache 2.0 in May 2026 and published three continually pretrained checkpoints in July 2026 — built on DNABERT-2, Nucleotide Transformer v3, and HyenaDNA. The checkpoints carry different terms from the code: they are distributed under the NVIDIA OneWay Noncommercial License, ready for non-commercial use only.

#Key Features

• Joint-embedding predictive objective: Predicts learned representations of masked genomic segments instead of raw nucleotides, emphasizing functional semantics over token recovery.
• Composite training signal: The JEPA cosine-similarity latent loss is combined with the backbone's native generative loss and VICReg-style variance and covariance regularization, with span-based masking and re-masking to prevent trivial recovery of the targets.
• Two-stage continual pretraining: The predictor is optimized first against a frozen backbone, after which predictor and backbone are optimized jointly.
• Released checkpoints with per-backbone data parity: Target-encoder weights for the DNABERT-2, NTv3, and HyenaDNA variants, each continually pretrained on its own backbone's original corpus so the gains reflect the objective and not fresh data.
• Measured benchmark gains: Mean AUROC gains across the 17 GFMBench-API tasks of about 2.6% for the DNABERT-2 backbone (+8.0% best task, +4.1% mean zero-shot), 2.2% for NTv3, and 3.3% for HyenaDNA.

#Technical Details

The target encoder is an exponential-moving-average copy of the backbone, and the predictor learns to map masked context onto its embeddings. Training minimizes a weighted sum of four terms — the backbone's own generative loss, the JEPA latent cosine-similarity loss, and VICReg variance and covariance penalties that keep the latent space from collapsing — first with the backbone frozen, then jointly.

The released checkpoints are the target-encoder weights. Each was continually pretrained on its own backbone's original data rather than on a shared corpus: the DNABERT-2 variant (a 117M-parameter BPE-tokenized encoder) on a ~4.76M-sequence subset of the DNABERT-2 multi-species corpus, the NTv3 100M variant on OpenGenome2 in random 8,192 bp crops (~384M records), and the HyenaDNA 16k variant on the hg38 human reference genome (~600K windows) — each with its own data loader and reproduction parameter file in the repository. Holding each run's data identical to its own baseline's is what isolates the JEPA objective for that backbone independently. The predictor is sized to the backbone too: a 4-layer Transformer in a reduced embedding space for DNABERT-2 and HyenaDNA, a 3-layer MLP over the pooled embedding for NTv3. Evaluation runs through the separate GFMBench-API library across 17 tasks — 9 supervised linear-probing and 8 zero-shot — reporting AUROC, AUPRC, and Matthews correlation coefficient. The repository provides run_jepa_pretrain.py with per-backbone parameter files, and run_benchmark.py accepts a downloaded .pt checkpoint via --checkpoint_path.

#Applications

The checkpoints are drop-in feature extractors: embeddings for linear probing, zero-shot embedding-based scoring of sequence perturbations, and retrieval-style comparisons over genomic regions, on tasks such as regulatory element classification and variant effect prediction. NVIDIA scopes them to research and representation learning and states they are not intended as diagnostic tools or clinical decision systems, and the noncommercial license rules out product use. The framework itself remains useful independently of the released weights: a group with its own DNA backbone or its own corpus can run the same continual-training stage over it.

#Impact

JEPA-DNA imports a representation-learning paradigm that reshaped vision and speech into genomics, arguing that predicting functional embeddings is a better objective for DNA than reconstructing tokens. Because each backbone was re-trained on its own original corpus, the comparison isolates the objective across three architectures and two generative losses — the part of the result most likely to transfer to backbones the authors did not test. With the code and three grounded checkpoints public, the claim is cheap for others to check, though the noncommercial terms mean a group can reproduce and extend the work far more easily than it can ship anything built on it. As a preprint, its conclusions await peer review.