stFormer

Single-cell transcriptomic foundation models learn contextual gene and cell embeddings from expression alone, treating each cell as an isolated bag of genes. Spatial transcriptomics adds the one thing that framing discards: where the cell sits, and therefore which neighbors are signaling to it. stFormer is a transformer built to use that information directly, encoding a cell's own transcriptome alongside the ligand genes expressed by cells in its spatial niche.

The model comes from the Institute of Process Engineering at the Chinese Academy of Sciences and the Institute of Image Processing and Pattern Recognition at Shanghai Jiao Tong University, first posted as a preprint in September 2024. Its central architectural move is a cross-attention layer inserted into each transformer block, letting genes in the center cell attend to niche ligand genes and so incorporating cell-cell communication into the representation itself rather than inferring it after the fact.

The practical obstacle is that spatial platforms trade resolution against gene coverage. Visium measures the whole transcriptome but each spot mixes several cells; imaging-based assays resolve single cells but over a restricted panel. stFormer addresses this with a biased cross-attention formulation that allows single-cell-resolution learning on deconvolved, spot-resolution Visium data — the most abundant public spatial resource.

#Key Features

• Ligand cross-attention over the spatial niche: Each transformer block adds a cross-attention layer that attends from center-cell genes to ligand genes of neighboring cells, encoding intercellular signaling into gene embeddings.
• Biased cross-attention for spot data: A derived correction lets the model train at single-cell resolution on whole-transcriptome Visium spots after cell type deconvolution, unifying the resolution/coverage tradeoff across platforms.
• Single-cell fallback mode: Disabling cross-attention while retaining layer normalization lets the same checkpoint process ordinary scRNA-seq, which is how its batch-correction behavior was benchmarked.
• In-silico ligand-receptor perturbation: Expression of specific ligands and receptors can be reduced computationally to read out downstream transcriptional consequences in neighboring cells.
• Cross-platform generalization: Although pretrained entirely on Visium, the model transfers to single-cell-resolution CosMx SMI data without retraining the backbone.

#Technical Details

stFormer stacks six identical transformer blocks, each combining self-attention, cross-attention, and a feed-forward layer, with 768-dimensional embeddings, 12 attention heads, and a feed-forward size of 3072. Gene symbols are looked up from a learnable 19,264-entry vocabulary initialized from scFoundation, and gene values pass through a scalar encoding module; only non-zero genes are input. The ligand gene library was assembled by intersecting seven knowledge databases and keeping genes present in at least two. The pretraining corpus was built by deconvolving 432 tissue sections from 75 human Visium datasets in the CROST repository with cell2location, yielding ~4.1 million cell-type-resolution spatial samples across diverse tissues, developmental stages, and disease states. Pretraining used a masked gene expression objective with MSE loss for two epochs, taking about ten days on four RTX 3090 GPUs; 0.5% of samples were held out and both losses followed power-law decay with sample size. On a normal myocardium section from a myocardial infarction Visium dataset, clustering of stFormer embeddings reached NMI 0.941 and ARI 0.961. For cell type prediction on a CosMx SMI human pancreas dataset, stFormer-4.1M reached 88.6% accuracy on a held-out field of view against scFoundation's 84.4%. Gene function prediction on TGF-beta and TNFA-via-NFKB gene sets gave median AUROC 0.90 and 0.93 versus 0.60 and 0.80 for scFoundation. An ablation pretrained on a 55-dataset subset (~2.4 million samples) consistently underperformed the full model, isolating the contribution of pretraining scale and diversity.

#Applications

stFormer serves spatial transcriptomics groups who want representations that already account for the microenvironment: annotating cell types in imaging-based spatial data, clustering and batch-correcting cells across sections and donors, predicting pathway membership for genes in a given tissue context, and screening ligand-receptor axes computationally before committing to perturbation experiments. The WNT signaling case study in ischemic myocardium illustrates the last of these, where combined downregulation of ligands and receptors in cardiomyocytes produced a stronger predicted effect than either alone.

#Impact

stFormer establishes that spatial context is worth encoding architecturally rather than appending as a post hoc analysis, and it does so on the most widely available spatial platform rather than requiring a specialized assay. Code, pretraining and fine-tuning scripts, and model weights are released under the MIT license, with weights, formatted spatial data, and case study inputs deposited on Zenodo — an unusually complete release for a model of this kind. Limitations are stated plainly: the pretraining corpus of ~4.1 million samples is far smaller than scFoundation's ~50 million, and stFormer initially trailed that baseline on clustering before fine-tuning closed the gap; the pretraining data itself is available only on request; and the work remains a preprint that has not been peer reviewed.