GeneJEPA

GeneJEPA is a self-supervised foundation model for single-cell transcriptomics that learns cell representations by predicting in latent space rather than reconstructing raw expression counts. Developed by Biostate AI and released as a preprint in October 2025, it adapts the Joint-Embedding Predictive Architecture (JEPA) — an approach that has driven progress in self-supervised vision and language modeling — to the transcriptome, framing the model as a predictive "world model" of cell state.

Most single-cell foundation models are trained to reconstruct masked or noisy gene expression values. GeneJEPA instead masks a subset of genes and trains the model to predict the latent representations of those held-out genes from the visible ones. Because the objective operates on learned embeddings rather than on noisy count values, the model is encouraged to capture the underlying structure of cellular state instead of fitting measurement noise, an approach intended to yield more transferable representations.

GeneJEPA is trained on Tahoe-100M, a large-scale single-cell RNA-seq atlas, and produces embeddings meant to transfer across tissues and datasets. Code is released on GitHub and pretrained weights are distributed on Hugging Face, positioning it as an openly available alternative in the crowded space of single-cell foundation models.

#Key Features

• Latent predictive objective: Predicts masked gene-set representations in embedding space rather than reconstructing expression counts, focusing learning on cellular structure rather than measurement noise.
• Perceiver-style set encoder: Encodes an unordered set of genes with fixed computational cost regardless of how many genes are provided, and supports test-time scaling by processing additional genes at inference.
• Continuous-value tokenizer: Combines gene identity with Fourier-feature encodings of expression, representing continuous expression levels without discretizing them into bins.
• Stable self-supervised training: Uses an exponential-moving-average teacher network with variance-covariance regularization to generate prediction targets, guarding against representation collapse.

#Technical Details

GeneJEPA pairs a student encoder with an EMA teacher: the teacher embeds a full or target gene set, the student predicts those target embeddings from a visible context set, and variance-covariance regularization keeps the learned embedding space informative and non-degenerate. Genes are represented as an unordered set processed by a Perceiver-style encoder whose cost is independent of input size, with each gene tokenized by combining a learned gene-identity embedding and Fourier-expanded expression features. The model is pretrained on the Tahoe-100M atlas, with data manifests pulled directly from the Hugging Face Hub, and it supports checkpoint-based inference. The authors report transferable embeddings that support downstream tasks including cell-type annotation, drug-response prediction, and perturbation reasoning.

#Applications

GeneJEPA is intended for computational biologists working with single-cell RNA-seq who need general-purpose cell embeddings that transfer to new tissues and datasets without task-specific retraining. Its representations can feed downstream analyses such as cell-type annotation, drug-response modeling, and reasoning about perturbation effects. The set-based encoder and test-time gene scaling make it convenient to apply to datasets with differing gene panels, and the openly released code and weights allow direct integration into existing single-cell workflows.

#Impact

GeneJEPA brings the joint-embedding predictive paradigm — which has reshaped self-supervised learning in other domains — to single-cell genomics, offering an alternative to the reconstruction objectives that dominate current transcriptome foundation models. By learning in latent space and training on the large Tahoe-100M atlas, it aims for more transferable cell representations, and its open release invites benchmarking against established single-cell models. As a preprint, the extent of its advantages over existing approaches will be clarified through independent evaluation on downstream tasks.