scRep

Sequence a cell twice and the two transcriptomes will disagree about which genes are present: limited capture efficiency, sequencing depth and technology-dependent coverage make every scRNA-seq profile a partial observation of the cellular state. Most single-cell foundation models are nevertheless trained to reconstruct masked expression values, which asks the network to model exactly that measurement variation rather than the biology beneath it.

scRep, developed at Jilin University and Beijing Zhongguancun Academy and posted to bioRxiv in September 2026, inverts the premise: it treats incompleteness as the training signal. For each cell the model builds several differently subsampled, gene-dropped and expression-masked views, then trains a student encoder to reproduce the latent representation a momentum-updated teacher assigns to a more complete view of the same cell. Nothing is reconstructed in observation space, so the only way to satisfy the objective is to encode structure that survives the perturbation. Distillation runs at two levels at once — the cell-level [CLS] state and the contextualized states of individual masked genes — so the same pretraining pass yields both a cell embedding and gene representations that depend on cellular context.

That places scRep alongside reconstruction-based models such as scGPT, Geneformer and scFoundation, and nearer to the view-alignment approaches of scConcept and GeneJEPA. Every reported result uses the frozen teacher encoder with no task-specific fine-tuning.

#Key Features

• Latent-space self-distillation: A student network matches prototype distributions produced by an exponential-moving-average teacher, replacing masked-expression reconstruction entirely — no auxiliary count-prediction loss is used.
• Asymmetric multi-view augmentation: The teacher sees one global view of up to 2,048 highly expressed genes; the student sees two global views and four local views of 256–768 genes, each subject to independent gene dropout.
• Joint cell- and gene-level objectives: Masked gene tokens are matched between teacher and student by gene identity rather than sequence position, giving contextual gene embeddings alongside the cell embedding.
• Order-invariant gene-set encoder: Cells enter as an unordered set of detected genes with no positional embeddings, so the encoder is insensitive to gene ordering.
• Diversity-aware corpus design: An abundance-weighted diversity index guides corpus construction, and a rebalanced 2M-cell corpus recovers the performance of a 2.8M-cell one.

#Technical Details

scRep is a 99.8M-parameter encoder-only Transformer: 12 pre-norm blocks, hidden dimension 768, 12 attention heads and a 3,072-dimensional feed-forward expansion. Its vocabulary covers 19,238 genes, and each cell's expression values are discretized into 50 equal-frequency bins computed independently within each view, so bins encode relative rather than absolute expression. Cell- and gene-level projection heads map onto 4,096 learned prototypes, with the teacher temperature fixed at 0.04 against a student temperature of 0.10 and teacher momentum annealed from 0.996 to 1.0. Two checkpoints of the same architecture were trained: scRep-3M on 2,803,535 cells drawn from Tabula Sapiens and CZ CELLxGENE, and scRep-30M on 30.72 million CELLxGENE profiles spanning 833 source datasets and 856 cell types.

Evaluation covers six held-out human datasets — glioblastoma, brain, kidney, eye, pancreas and heart, 684,233 cells in total — all excluded from the pretraining corpora. Across them scRep reaches a mean normalized mutual information of 0.859 and adjusted Rand index of 0.821 under Leiden clustering calibrated to the annotated cell-type count, ahead of STATE, STACK, scConcept, scFoundation, scGPT and Geneformer-v2. For k-nearest-neighbour label transfer across donor-disjoint folds, scRep attains a mean macro-F1 of 0.8752 with the full reference set and 0.7222 when only 0.5% of reference cells per type are retained — a drop of 0.1530, the smallest among the models compared.

#Applications

The frozen embeddings serve the routine steps of a single-cell study without retraining: unsupervised clustering, transfer of cell-type labels from a small annotated reference, and trajectory analysis. Around 88% of curated marker genes rank above expression- and frequency-matched controls under a perturbation-based importance score, giving an interpretable readout of which genes drive a cell's representation, while contextual gene embeddings organize transcription factors with candidate targets into cell-type-specific programs. Graph-based diffusion pseudotime over the same representations recovers developmental ordering in human peri-implantation embryos (Spearman ρ = 0.794 against collection day) and separates lineage-restricted progenitors from haematopoietic stem cells at an AUROC of 0.959 without using the FACS labels.

#Impact

scRep's most portable result is about corpora rather than architecture: a 2.0M-cell corpus rebalanced for cell-type coverage matched a 2.8M-cell one built by straightforward accumulation, indicating that redundant cells from well-represented populations contribute little and that curation can substitute for volume. Both checkpoints and an inference implementation are published on Hugging Face under Apache 2.0, though the scripts reproducing the paper's analyses are promised only on publication. The work remains a preprint awaiting peer review, its benchmarks are confined to human tissues profiled largely on 10x Genomics platforms, and the transcription-factor networks it recovers are embedding-similarity associations rather than validated regulatory interactions.