GenoJEPA

GenoJEPA is a genomic foundation model that reframes self-supervised pretraining on DNA as a problem of semantic prediction in latent space rather than reconstruction of individual nucleotides. Most genomic language models, including DNABERT-2 and the Nucleotide Transformer family, inherit objectives from natural-language processing and treat DNA as a string of tokens to be masked and reconstructed. The authors argue that this framing is poorly matched to genomes: DNA lacks the explicit semantic boundaries of written language and carries substantial evolutionary noise, so forcing a model to reconstruct exact bases in a low-dimensional input space can waste capacity and yield representations with limited discriminative power.

To address this, GenoJEPA adapts the joint-embedding predictive architecture (JEPA) — originally developed for images and video — to genomic sequence. Instead of predicting masked bases, the model predicts the latent representations of masked regions from the representations of visible context, aligning embeddings semantically rather than at the nucleotide level. The work was developed by researchers at the Beijing University of Posts and Telecommunications and posted to bioRxiv in April 2026.

The paper positions GenoJEPA as a parameter-efficient alternative to larger masked-language genomic models, reporting competitive or superior performance with roughly an order of magnitude fewer parameters, and validating the quality of its frozen features through lightweight, GPU-free classifiers.

#Key Features

• Latent-space predictive objective: Rather than reconstructing nucleotides, GenoJEPA predicts the embeddings of masked sequence regions from visible context, shifting optimization from base-level reconstruction to semantic alignment.
• Continuous patching: Inspired by Vision Transformers, the model splits a DNA sequence into non-overlapping patches (e.g., 16 nucleotides) and linearly projects each into a continuous vector, avoiding discrete k-mer or BPE tokenization.
• Multi-view augmentation: Pretraining uses global views (roughly 65–80% of a sequence) and local views (roughly 35–40%), encouraging representations that are robust across scales.
• Parameter efficiency: Two variants — GenoJEPA-T (~6M parameters) and GenoJEPA-B (~52M parameters) — match or exceed baselines that are several times larger.
• Frozen-feature usability: Embeddings from a fixed checkpoint feed simple classifiers such as logistic regression, enabling competitive downstream accuracy without GPU fine-tuning.

#Technical Details

GenoJEPA couples a Transformer encoder over continuous DNA patches with a JEPA-style predictor and an exponential-moving-average target encoder, and applies a latent-space regularizer to prevent representational collapse and structure the embedding space. The models are pretrained with a context length of 4,096 base pairs on a multi-species genomic corpus spanning 850 species, the same dataset used by Nucleotide Transformer v2. Across 55 downstream tasks drawn from the Genomic Benchmarks, GUE, and Nucleotide Transformer task suites, GenoJEPA-B reports an average linear-probing Matthews correlation coefficient of about 0.589, exceeding NT-v2 (~494M parameters, ~0.519) and DNABERT-2 (~117M parameters, ~0.529) despite using only ~52M parameters; under fine-tuning it reports an average MCC near 0.704. The authors also report that performance degrades gracefully in few-shot settings, remaining strong when only 10% of labeled data is available.

#Applications

GenoJEPA is intended as a general-purpose embedding model for genomic sequence analysis. Its frozen representations can be applied to regulatory-element classification, promoter and enhancer detection, splice-site prediction, epigenetic-mark and chromatin-state classification, and other sequence-labeling tasks covered by standard genomic benchmarks. Because high-quality features are available without fine-tuning, the model is particularly useful for research groups with limited compute, who can run lightweight CPU-based classifiers on extracted embeddings rather than fine-tuning large transformers on GPUs.

#Impact

GenoJEPA contributes to a growing line of work exploring whether predictive, non-generative self-supervised objectives can outperform masked-language modeling for biological sequences, joining contemporaneous efforts such as JEPA-DNA. By demonstrating that latent-space prediction can match or surpass substantially larger masked-language genomic models, it strengthens the case that architectural and objective design — not raw parameter count — drives representation quality in genomics. Its main limitations are a relatively short 4,096 bp context, which restricts modeling of long-range interactions such as topologically associating domains, and an unexplored scaling regime, since the largest reported variant is only ~52M parameters. As a recent preprint, its real-world adoption and independent validation remain to be established.