scMomer

Single-cell foundation models have overwhelmingly been trained on one assay at a time, because that is where the data is: scRNA-seq and scATAC-seq atlases are large, while genuinely paired multi-omic profiles of the same cells remain scarce. The result is a mismatch. A model trained only on transcriptomes cannot reason about chromatin accessibility, yet the vast majority of experiments a biologist actually has in hand are exactly that — one modality, not two.

scMomer, from Macao Polytechnic University with collaborators at the University of Electronic Science and Technology of China, Tianjin University, and Shandong University, is built around that asymmetry rather than against it. Its premise is that a model can be taught what the missing modality would have looked like, then asked to produce multi-omics-aware embeddings from whatever single modality is available. This is achieved through a three-stage curriculum: self-supervised pretraining on each modality separately, joint fusion on paired data, and knowledge distillation into a compact student that reconstructs the absent modality's representation.

The resulting embeddings are evaluated with the backbone largely frozen across an unusually wide task list — cell type annotation, cross-modality translation, gene function prediction, drug response prediction, and perturbation prediction — which is the pattern that distinguishes a reusable representation from a task-specific model.

#Key Features

• Missing-modality robustness by design: A lightweight cross-modality bridge predicts ATAC embeddings from RNA input alone, so multi-omics-aware representations can be produced for the far more common unimodal datasets.
• Three-stage pretraining curriculum: Unimodal masked modeling, then multimodal fusion with a pairing discriminator, then distillation — each stage targets a different level of structure rather than optimizing one joint objective.
• Modality-specific encoders: RNA and ATAC are handled by architectures suited to their dimensionality, and the modular design allows either backbone to be swapped for a stronger one.
• Frozen-backbone downstream use: Most parameters stay fixed while a small MLP head is trained per task, and the embeddings still outperform unimodal baselines across cell-, gene-, and drug-level benchmarks.
• Robustness to class imbalance and small data: On rare-cell-type and downsampled variants of the Zheng68K PBMC benchmark, performance degrades more gracefully than the unimodal baseline.

#Technical Details

The RNA encoder is built on pretrained scBERT, whose Performer blocks scale attention across thousands of genes per cell; its output is projected through a 1D convolution and linear layer into a cell embedding. The ATAC encoder addresses the

100,000-peak dimensionality by segmenting the genome into fixed-length positional patches in the manner of a vision transformer, adding 1D positional encodings, and passing them through a Performer encoder. Stage I trains both with masked modeling — cross-entropy over binned expression for RNA, squared error over patch accessibility for ATAC. Stage II concatenates the two embeddings through an MLP fusion network trained with dual reconstruction losses plus a binary modality-matching discriminator. Stage III freezes the RNA encoder and trains a feedforward student to regress the teacher's ATAC embedding. Pretraining uses a pseudo-paired fetal multi-organ atlas of 377,134 cells (47 cell types, 36,601 genes, 1,154,464 ATAC peaks) and a 3,233-cell paired human brain dataset. On Zheng68K, scMomer reaches 0.808 accuracy and 0.703 macro F1 against scBERT's 0.768 and 0.647; under 100:1 rare-type imbalance it holds 0.838 accuracy and 0.829 macro F1. Gene-level property prediction gives AUC 0.787 against scBERT's 0.766, and integrating scMomer gene embeddings into GEARS improves perturbation-response prediction on the Adamson and Norman Perturb-seq datasets.

#Applications

scMomer is aimed at groups holding large unimodal single-cell datasets who want representations informed by regulatory structure they did not measure. Concretely: cell type annotation on imbalanced or small cohorts, in-silico generation of chromatin accessibility profiles from RNA-only experiments, transfer of embeddings into bulk drug response frameworks such as DeepCDR, and gene-level function prediction. The frozen-backbone plus MLP-head pattern keeps the compute cost of adopting it low.

#Impact

scMomer is a clear statement that missing modalities should be treated as the default condition in single-cell modeling rather than an edge case, and that distillation is a practical route to honoring that. Its modular encoder design invites replacement of either backbone as stronger unimodal models appear. The work is a preprint that has not been peer reviewed, the preprint carries no reuse license and states no code availability, and evaluation rests on a pseudo-paired multi-omics corpus in which cells are matched by annotated cell type rather than physically co-assayed. Code is distributed separately from the preprint under GPL-3.0, while the pretrained checkpoint is served from a linked cloud folder outside the repository and carries no license of its own.