scMMGPT

scMMGPT is a multimodal foundation model that learns single-cell representations from gene expression and natural-language text at the same time. Single-cell language models such as scGPT, Geneformer, and scBERT are pretrained purely on transcriptomes, so the biological knowledge accumulated in ontologies, marker-gene catalogs, and the literature never reaches them. scMMGPT injects that knowledge by aligning each cell's expression profile with prose describing what the cell is, developed by researchers at the University of Science and Technology of China, the Harbin Institute of Technology, and the National University of Singapore and released as a March 2025 preprint.

Two design choices separate it from earlier cell-text models. First, it keeps quantitative expression values: rather than serializing a cell into a ranked list of gene names in the manner of Cell2Sentence, it feeds the profile through a pretrained single-cell transformer, so magnitude information survives into the shared representation. Second, it connects the cell encoder and the language model in both directions through learned projectors, letting text condition cell generation as well as the reverse.

Pretraining runs in two stages: a discriminative stage that aligns the two modalities contrastively, then a generative stage that trains the model to write a cell's description and to produce an expression profile from a description. The result transfers to cell-type annotation, clustering, and — the setting the authors emphasize — labeling cell types that never appeared in pretraining.

#Key Features

• Bidirectional cell-text projectors: A Q-Former with 32 learned queries maps cell embeddings into the language model's token space, while a lightweight projector turns text embeddings into soft prompts that condition the cell encoder.
• Expression-preserving cell encoding: The cell branch consumes gene tokens paired with their measured expression values, avoiding the information loss of rank-based "cell sentence" serialization.
• Two-stage pretraining: Contrastive and matching objectives establish alignment first; cell-to-text captioning and text-to-cell reconstruction then push semantic structure into both encoders.
• Open-vocabulary annotation: Because cell types are represented as text, the model scores a profile against arbitrary type descriptions and can label categories held out of its training corpus.
• Cell description generation: Given a profile, the model writes a paragraph naming and characterizing the cell, which doubles as an interpretable annotation output.

#Technical Details

The cell branch is scGPT, operating over a 60,697-gene vocabulary with the 2,048 highest-expressed genes retained per cell; the text branch is LLaMA-2 7B. The cell-to-text Q-Former is initialized from BiomedBERT, and the text-to-cell path is an MLP that produces soft prompts for scGPT. Stage 1 trains the cell encoder and Q-Former for five epochs with contrastive and cell-text matching losses; stage 2 adds cell-to-text captioning and a mean-squared-error text-to-cell objective for one epoch, with LoRA adapters on the language model. Pretraining used approximately 27 million human cells drawn from 344 CELLxGENE datasets (filtered down from 60.5 million profiles), paired with text assembled from cell metadata and free-form ontology and encyclopedia descriptions. On fine-tuned cell-type annotation the model reports 69.0% accuracy on Myeloid, 98.2% on hPancreas, 87.4% on Multiple Sclerosis, and 94.8% on PBMC-3K, with macro-F1 gains of roughly 10 points over the strongest baseline on the latter two. Clustering biological-conservation scores reach 0.854 on PBMC-10K and 0.545 on COVID-19. On Tabula Sapiens, whose 161 cell types are largely absent from pretraining, it reaches 49.1% zero-shot top-1 accuracy, rising to 60.9% after fine-tuning on 30% of the type set.

#Applications

The model suits atlas-scale annotation work where the label space is open: mapping a new dataset onto cell types that no fixed classifier head covers, harmonizing labels across studies that use different nomenclature, and clustering while correcting batch effects. The description-generation head gives biologists a readable justification alongside a label, and the text-to-cell direction supports querying what a described population's expression profile should look like.

#Impact

scMMGPT argues that the ceiling on single-cell representation learning is not only transcriptomic data volume but the biological context that lives in text, and its out-of-distribution results are the strongest evidence for that claim: open-vocabulary matching handles novel cell types that a closed-set classifier structurally cannot. The published repository contains both pretraining stages and the model code, but it carries no license file and no released checkpoint, so reproduction currently requires rerunning pretraining from scratch. The work remains a preprint, evaluation is confined to human tissue, and the authors name extension to other modalities such as scATAC-seq and CITE-seq as future work.