EpiFoundation

EpiFoundation is a self-supervised foundation model for single-cell ATAC-seq (scATAC-seq) data, developed by researchers at Columbia University, UC Santa Cruz, and Duke University and released as a preprint in February 2025. Single-cell foundation models have concentrated almost entirely on transcriptomics — scGPT, Geneformer, scBERT, and scFoundation all learn from scRNA-seq — leaving chromatin accessibility, the assay that reads out the regulatory state of individual cells, served mainly by task-specific methods.

Porting the scRNA-seq recipe directly does not work. A scATAC-seq experiment yields 10^5 to 10^6 peaks, and because a diploid cell carries only two copies of each locus, the matrix is extremely sparse — making masked-token pretraining over that space both computationally prohibitive and statistically weak.

EpiFoundation resolves this with two design choices. First, it feeds the model only the non-zero peak set of each cell, which raises input information density and caps sequence length independently of the total peak vocabulary. Second, rather than reconstructing masked peaks, it supervises pretraining with the paired transcriptome: the model predicts which genes are expressed in the same cell, forcing the cell embedding to align peak-level accessibility with gene-level phenotype. To enable this the authors curated MiniAtlas, a 10x Multiome corpus of more than 100,000 cells with matched scRNA-seq and scATAC-seq spanning 19 tissues and 56 cell types.

#Key Features

• Non-zero peak tokenization: Each cell is represented as the set of its accessible peaks, sampled to a maximum of 12,000 tokens — enough to retain every non-zero peak for more than 95% of cells — instead of embedding the full 10^5–10^6 peak vocabulary.
• Peak-to-gene alignment objective: Pretraining predicts binarized expression over 8,000 sampled genes per cell, balanced between expressed and silent genes, using cross-entropy loss. Cross-modality supervision replaces the masked-reconstruction objective used by scRNA-seq foundation models.
• Chromosome-aware embeddings: Every peak token is summed with an embedding of its parent chromosome (22 autosomes plus X and Y), injecting coarse genomic locality into an otherwise unordered input set.
• Batch disentanglement at pretraining time: A separate batch embedding is concatenated to the cell embedding during pretraining only, so the representation extracted at inference is free of the batch covariate.
• Open weights, code, and training corpus: The pretrained checkpoint and MiniAtlas h5ad files are distributed on HuggingFace under Apache 2.0, alongside PyTorch training and evaluation code.

#Technical Details

The encoder is a 6-layer transformer with 8 attention heads and a 512-dimensional embedding, built on FlashAttention-2, with a [CLS] token pooled into the cell representation. Training ran for 140 epochs with Adam, a 1e-4 learning rate under cosine annealing, and dropout 0.15. MiniAtlas was assembled from GEO and ENCODE FASTQ files processed with Cell Ranger ARC 2.0.1 against GRCh38, with peaks called jointly by MACS2 and cell types assigned by Spearman correlation to DISCO reference profiles.

Downstream tasks are handled by lightweight decoders fine-tuned on the frozen pretraining backbone. On cell-type annotation, EpiFoundation reaches 0.9135 accuracy on kidney, 0.8837 on PBMC, 0.7615 on BMMC, and 0.8423 on the combined ALLTissue set, with ROC-AUC between 0.976 and 0.987; macro F1 trails accuracy substantially (0.50–0.71), reflecting weaker performance on rare cell types. For batch correction it leads PCA, Harmony, LIGER, and scANVI on most metrics — NMI of 0.5681 on kidney versus 0.3273 for the best baseline — though PCA retains an edge on isolated-label score for kidney and PBMC. On gene expression prediction it roughly doubles the Spearman correlation of Signac's Gene Activity heuristic (0.4221 versus 0.1609 on PBMC). Ablations confirm both design choices contribute: removing the batch embedding drops kidney NMI to 0.4695, and removing chromosome embeddings drops it to 0.4354.

#Applications

EpiFoundation targets the analytical bottlenecks that follow a scATAC-seq experiment: annotating cell types directly from accessibility profiles without a paired transcriptome, integrating samples collected across batches or laboratories, and imputing per-cell gene activity in assays where only chromatin was measured. For groups studying regulatory heterogeneity in development, differentiation, or disease progression, the pretrained cell embedding provides a transferable starting point that replaces the per-dataset dimensionality reduction pipelines built on LSI or PCA.

#Impact

EpiFoundation is among the first models to demonstrate that the foundation-model paradigm transfers to single-cell epigenomics, and its cross-modality supervision offers a template for pretraining on sparse assays wherever a denser paired modality exists. The release of MiniAtlas as a curated, uniformly processed paired corpus is a contribution in its own right, since assembling comparable multiome data has been a barrier for the field. Practical constraints remain: the model is human-only and tied to GRCh38 with a fixed peak vocabulary, so new data must be processed against the same peak set; each downstream task needs supervised fine-tuning rather than zero-shot inference; and pretraining depends on paired multiome data, which is far scarcer than scATAC-seq alone. The work remains a preprint awaiting peer review.