scEMB

A cell profiled on 10x Genomics and the same cell type profiled by single-nucleus sequencing produce count matrices that disagree on almost every absolute number, yet a model trained on one should transfer to the other. Rank-based encodings solve half of that problem by throwing away the counts entirely; expression-value encodings keep the counts but inherit the platform's scale. scEMB, a transformer developed at UTHealth Houston with collaborators at the MD Anderson Cancer Center, takes the middle path: it normalizes each cell to counts per million, log-transforms, then bins genes into 100 intervals ordered by their real expression values, and engineers the positional embeddings to preserve that order.

The binning strategy is adapted from scGPT, while the masked pretraining objective and the rank-ordered treatment of a cell's transcriptome follow Geneformer. What scEMB adds is a pair of downstream analyses built on the embedding space rather than on a task head: an in silico perturbation test that scores how far a masked gene moves a cell's embedding, and an in silico correlation analysis that asks which gene perturbations move cells along the same axis as a disease state transition. The second is the one aimed at therapeutic discovery — a perturbation that reverses the disease direction is a target hypothesis.

scEMB was posted as a bioRxiv preprint in September 2024 by Kang-Lin Hsieh, Yan Chu, Xiaoyang Li, Patrick G. Pilié and Yulin Dai, and has not been peer reviewed.

#Key Features

• Ordered expression binning: Genes are binned into 100 intervals ranked by real expression value rather than by rank alone, so gene-expression hierarchy and cell-type specificity survive integration across sequencing platforms.
• Embedding-space perturbation scoring: Masking a CRISPR-targeted gene in control cells and measuring cosine similarity against the actual perturbed cells turns a 768-dimensional embedding into a readout of how consequential that gene is.
• In silico correlation analysis: Cell embeddings are concatenated with gene perturbation embeddings from a GO- and PPI-guided graph attention network, letting the model compare a disease state transition against the effect of perturbing any given gene.
• Two fine-tuning routes: LoRA realigns the model's own layers to a new dataset, while prefix tuning aligns an external knowledge vector to it — the choice depends on how far the downstream task sits from the pretraining objective.
• Length-grouped dynamic padding: Sorting minibatches by transcriptome length before padding cut wasted computation enough to give a 29.4× pretraining speedup.

#Technical Details

scEMB is a BERT-style encoder of twelve transformer blocks, each with a self-attention layer and a feed-forward layer, using 768-dimensional embeddings, 12 attention heads, a feed-forward size of 3,072 and an input length of 2,048 tokens — long enough to hold the genes typically detected in one cell under full attention, accelerated with scaled dot-product attention. Pretraining used masked language modeling with 15% of genes masked per transcriptome, the Adam optimizer with weight decay, 10,000 warmup steps and DeepSpeed-partitioned distributed training; it took roughly ten days on a single node of eight NVIDIA H100 96GB GPUs.

The pretraining corpus was drawn from the CZ CELLxGENE Discover Census in May 2024, covering both scRNA-seq and snRNA-seq protocols from healthy samples, filtered to remove cells with fewer than 200 detected genes or over 10% mitochondrial content — 30 million cells in total. On zero-shot clustering and batch integration over the PBMC 10k dataset, scEMB was benchmarked against Geneformer, scGPT and a conventional Scanpy pipeline across isolation, kBET and batch-mixing-entropy scores, with the authors reporting performance on par with the two foundation-model baselines. Cell type annotation was evaluated by fine-tuning on one SEA-AD middle temporal gyrus sample and testing on a second from a different batch.

#Applications

The perturbation work is the practical draw. Validated against a CRISPRi screen in iPSC-derived microglia covering 39 single-sgRNA gene conditions, scEMB's embedding-similarity ranking recovered microglial regulators including CSF1R and TGFBR1/2 among its top predictions. Applied to microglia from Alzheimer's disease and cognitively normal donors in the ROS/MAP cohort, the in silico correlation analysis placed the GWAS risk genes PLCG2, SORL1 and TREM2 among the top-scoring perturbations, the kind of prioritization that decides which genes go into an expensive genome-scale screen.

#Impact

scEMB is a careful re-examination of design choices in single-cell foundation models rather than a departure from them, and its value lies in what it makes routine: comparing a disease state transition and a genetic perturbation in one shared embedding space. The authors are explicit that zero-shot perturbation predictions still need care, since most perturbation conditions never appear in pretraining, and they present the fine-tuned correlation analysis as the more trustworthy route. Practical uptake is constrained by release: the preprint publishes no code repository and no checkpoint, and its data availability statement offers the analyzed data from the authors on request.