EpiAgent

Single-cell foundation models have concentrated on the transcriptome — scGPT and Geneformer both learn from gene expression. Chromatin accessibility is arguably the more decisive readout of cell state, since it describes the regulatory landscape that expression follows from, but scATAC-seq resists the same treatment. A human genome carries on the order of a million candidate cis-regulatory elements, only a small fraction are accessible in any one cell, and the signal is close to binary. Methods built on cell-by-cCRE matrices therefore have to filter features heavily before they can compute anything, throwing away accessible elements in the process.

EpiAgent takes a different representation. For each cell it keeps only the accessible cCREs, ranks them by a global TF-IDF importance weighting, and writes them out as a "cell sentence" — a variable-length token sequence, analogous to a document, with the ranking playing the role of positional information. Bidirectional attention over that sentence produces both a cell embedding and contextualized embeddings for each element. The model was developed by Xiaoyang Chen, Rui Jiang and colleagues at Tsinghua University, posted to bioRxiv in December 2024 and published in Nature Methods in 2025.

Within the catalog it sits alongside other scATAC foundation models such as ChromFound and Atacformer, differing in its sentence-style tokenization and in the breadth of downstream tasks driven from a single pretrained checkpoint.

#Key Features

• Cell sentences: Only accessible elements are tokenized, so input length scales with what a cell actually shows rather than with the size of the reference cCRE set, and no feature pre-filtering is needed.
• Zero-shot annotation: Two supervised variants, EpiAgent-B for brain and EpiAgent-NT for other tissues, label cells in newly sequenced datasets directly, without fitting anything on the new data.
• Perturbation tokens: A learnable token appended to the classification token encodes a stimulus; for genetic perturbations, gene embeddings passed through a graph neural network over Gene Ontology terms let the model handle interventions it never saw in training.
• In-silico cCRE knockout: Because a cell is a list of elements, removing one is a direct edit to the input — the resulting shift in cell embedding quantifies that element's contribution to cell state.
• Reference mapping: Batch-specific token embeddings let query datasets be projected onto an integrated reference without retraining the encoder per dataset.

#Technical Details

Pretraining used the Human-scATAC-Corpus, assembled from 28 public datasets spanning 31 tissues and cell lines: approximately 5 million cells and 35 billion tokens. The model has roughly 1.4 billion parameters in three parts — a cCRE embedding module of about 695 million, an 18-layer transformer encoder of about 57 million using FlashAttention, and a single-layer signal decoder of about 695 million that reconstructs accessibility across all elements from the cell embedding. Two objectives drive pretraining: a cell-cCRE alignment task, in which the model classifies whether sampled elements are accessible in the cell, and a signal reconstruction task through the decoder. On supervised cell-type annotation across benchmark datasets, EpiAgent exceeds the second-best method by an average of 11.036% in accuracy, against CellCano, and 21.549% in macro F1, against SANGO. Evaluation datasets were deliberately chosen to span similarity to the corpus, from heart and brain data with overlapping cell populations to hematopoietic differentiation and in-vitro stem cell data whose cell types differ substantially from anything pretrained on.

#Applications

The practical draw is annotation of new scATAC-seq experiments without assembling a labeled reference first, which is where most chromatin-accessibility projects stall. Beyond that, imputation recovers accessibility in sparse cells, reference integration folds new samples into an existing atlas, and the perturbation head predicts the direction of accessibility change for differentially accessible elements under stimuli or genetic edits not present in training. The knockout capability was demonstrated as an in-silico treatment experiment: promoters of ABCC1, VEGFA and EGLN3 were deleted from cancer cell sentences and the shift was scored against a synthetic reference of chimeric cells with graded cancerization, showing larger movement toward a normal-like state than random element knockouts produce.

#Impact

EpiAgent establishes that the language-model framing transfers from transcriptome to epigenome once the tokenization is chosen to match the data's sparsity rather than fighting it, and that a single checkpoint can carry feature extraction, annotation, imputation, perturbation response and reference mapping. Code is on GitHub under an MIT license, an epiagent package is installable from PyPI with demo notebooks for each task, and the Human-scATAC-Corpus has since been published as a standalone resource. Two limits are worth naming: the pretrained model weights are distributed through a Google Drive folder rather than a versioned model hub, and the corpus is human-only, so cross-species transfer is unaddressed. Performance also degrades, as expected, on datasets whose cell types are furthest from the pretraining distribution.