scDMC

Single-cell language models must turn a cell's transcriptome — an unordered vector of tens of thousands of gene counts — into a token sequence, and nearly all commit to one encoding scheme. Geneformer rank-orders genes by normalized expression, discarding magnitude; scGPT and scBERT bin or embed expression values, weakening the relative-importance signal rank encoding captures. Either choice loses information before the model sees the cell.

scDMC (single-cell Dual-stream Masked Contrastive pre-training) refuses to choose. Developed by Honglie Guo, Shunfang Wang, and colleagues at Yunnan University and posted to bioRxiv in September 2025, it builds two parallel token sequences per cell — a rank-based stream encoding which genes matter most relative to one another, and an expression-aware stream preserving each gene's signal magnitude — and feeds both through weight-sharing encoders. Two self-supervised objectives run on top: dual-stream masked language modeling learns gene-level co-expression context, while cell momentum contrastive learning treats the two streams of one cell as a natural positive pair and pulls their [CLS] embeddings together against a queue of other cells.

The authors frame scDMC as a data-efficiency argument. Pretrained on 2 million cells — an order of magnitude fewer than scFoundation or scGPT — it reports leading scores on cell-type annotation, clustering, and atlas-level batch integration, alongside interpretability analyses that recover functional gene modules and cell-type-specific regulatory networks. Code, weights, and the tokenized pretraining corpus are all publicly released.

#Key Features

• Dual-stream input encoding: Each cell is tokenized twice, by expression rank and by expression magnitude, so the model retains both the functional ordering of genes and their absolute abundance instead of trading one for the other.
• Cell momentum contrastive learning: A MoCo-style objective with an EMA momentum encoder (coefficient 0.995) uses the rank and expression views of a cell as positive pairs, yielding a discriminative cell embedding space without labels or synthetic augmentation.
• Interpretable gene embeddings: Clustering the contextual gene embeddings recovers coherent functional modules — adaptive immunity and ribosomal programs in CD8+ T cells, protein homeostasis and stress response in hematopoietic stem cells — and supports inference of regulatory networks.
• Long-sequence efficiency: FlashAttention and ALiBi positional biases replace standard attention and learned position embeddings, handling up to 2,048 gene tokens while suiting the non-sequential nature of gene order.

#Technical Details

scDMC is a BERT-style masked-language encoder: 12 transformer layers, 512-dimensional hidden states, 8 attention heads, 1,024-dimensional feed-forward width, and a maximum sequence length of 2,048 — roughly 63 million parameters in the released float32 checkpoint. The shared vocabulary holds 60,534 tokens: 60,530 genes keyed by Ensembl ID plus [PAD], [MASK], [CLS], and [EOS], so heterogeneous studies share one token space. The framework runs two weight-sharing online encoders plus a momentum encoder whose parameters are an exponential moving average of the online weights; the loss is a weighted sum of the dual-stream MLM and CellMoCo terms.

Pretraining used 2 million transcriptomes from healthy human blood in the CELLxGENE Discover Census release of 2024-07-01, with no quality-control filtering or highly variable gene selection, over three epochs and 250,002 optimizer steps. On the imbalanced Zheng68K benchmark scDMC reaches 86.60% annotation accuracy and a macro F1 of 0.767, 2.15% above the next-best model; reported macro F1 gains are 5.18% on BMMC, 2.21% on MS, and 0.20% on PBMC10K. Leiden clustering on its embeddings gives the highest average ARI and NMI across six datasets, and scIB integration scoring places it ahead of the compared baselines on BMMC (12 donor batches) and Immune (10 platform batches). Ablations from rank-only to expression-only to dual-stream to full scDMC show each component contributing, with the contrastive term adding the largest increment.

#Applications

The immediate uses are standard single-cell workflow tasks: annotating cell types by fine-tuning the encoder with a linear classification head, clustering cells from frozen embeddings, and integrating experiments across donors and platforms for atlas construction. Fine-tuned checkpoints for Zheng68K and PBMC10K ship alongside the pretrained model, giving a runnable starting point. The gene embeddings also support hypothesis generation — grouping genes into functional modules and identifying hub genes in cell-type-specific networks — for immunologists and hematologists working with blood and bone marrow data.

#Impact

scDMC demonstrates that input-encoding design, not corpus size alone, drives single-cell representation quality — a counterweight to the scaling emphasis of most single-cell foundation models. Its openness is unusually complete for the field: source code, the pretrained and fine-tuned checkpoints, and the tokenized 2-million-cell corpus are all downloadable, making the efficiency claim independently checkable. Several caveats bound it. The preprint has not been peer reviewed. Pretraining covers only healthy human blood, leaving generalization to solid tissues, disease states, and other species untested; the authors flag cross-tissue pretraining as future work. The model is transcriptome-only, with CITE-seq and ATAC-seq streams proposed rather than implemented. All baseline comparisons were run by the authors, and the GitHub repository carries no license file, leaving reuse terms unstated.