CellPatch

A single-cell RNA-seq profile can span tens of thousands of genes, and most transformer models for single cells assign one token per gene. That convention makes the input sequence long enough that attention cost, rather than biology, sets the ceiling on what a model can read — which is why single-cell transformers such as scBERT and scGPT discretize expression values and limit how many genes enter the sequence. CellPatch attacks the sequence length itself instead of working around it.

The mechanism is a set of 64 learned patch tokens sitting in front of the encoder. Each patch token queries the cell's expressed genes through a cross-attention block, so an arbitrary number of measured genes collapses into a fixed 64-vector summary of the cell — the paper's "heuristic patching". Self-attention then runs over those 64 patch features, so the quadratic term applies to 64 elements rather than to the gene list. Because the compression is learned during pretraining rather than imposed by a gene filter, each patch behaves as a meta-gene: a recurring, data-derived combination of genes the model reuses across cell types.

CellPatch was developed by Hua-Jun Wu's group at Peking University Health Science Center with collaborators at Shanghai Normal University, Shanghai Jiao Tong University School of Medicine, and two Peking University hospitals, and posted to bioRxiv in November 2024.

#Key Features

• Fixed-width cell representation: Every cell becomes the same patch feature matrix regardless of how many genes were detected, so downstream heads need no re-sizing for a new dataset or sequencing platform.
• Gene embeddings used twice: The same learned gene embeddings act as positional encodings in the encoder and as query prompts in the decoder, where they retrieve each gene's expression value back out of the patch features.
• Transfer into other tools: The patching module can be spliced into existing single-cell software rather than replacing it, demonstrated by feeding patch features into STAGATE's graph autoencoder for spatial domain detection.
• Readable attention: Cross-attention weights from the patching step are cell-type specific — CD8+ cytotoxic T cells put high weight on IL32 — and the learned gene embeddings cluster functionally, placing PF4 and PPBP near CD34+ marker genes.

#Technical Details

The encoder is one cross-attention layer over 64 patch tokens followed by two self-attention layers, at an embedding dimension of 32 with 2 attention heads; the pretraining decoder is one cross-attention and one self-attention block reconstructing a scalar expression value per gene prompt. Gene identity is tokenized against a 60,690-gene reference built from CELLxGENE, with room for 70,000 tokens, and sequences are capped at 3,000 genes for encoding and 1,000 for decoding. Pretraining used masked gene modelling — 30% of non-zero values masked, mean squared error on the reconstruction — over 10 million cells sampled from CELLxGENE, for 50 epochs at batch size 512 with Adam. Inputs are normalized to 10,000 counts per cell, log1p transformed, and clipped at 5 following scBERT.

On cell type annotation, the fine-tuned model reports higher accuracy and F1 than scBERT, scGPT (both fine-tuned and trained from scratch) and LangCell across the Zheng68K, Pancreas, Myeloid and Segerstolpe datasets. For spatial transfer, patch features concatenated into STAGATE raised the adjusted Rand index on DLPFC slice 151675 (3,592 spots, 33,538 genes) from 0.59 to 0.65, with improvement across all twelve sections. Inference runs about 1,000 cells in 2 seconds on an A100 80GB.

#Applications

The direct use is cell type annotation on new scRNA-seq datasets, where the released checkpoint is either fine-tuned or held frozen behind a linear probe. The more distinctive use is as a component: because the patching module emits a compact, fixed-width feature per cell, it can be concatenated into the input of an existing analysis method — the STAGATE integration is the worked example — to give a tool that had none some pretrained transcriptomic context. The small footprint also puts pretrained single-cell representations within reach of groups with one GPU rather than a cluster.

#Impact

CellPatch argues that single-cell foundation models have been scaling the wrong axis: its contribution is a tokenization scheme rather than a larger corpus or parameter count, and the 10-million-cell pretraining run is modest beside the roughly 100-million-cell corpora behind models such as CellFM. The authors are explicit that more training iterations and a broader sample space would improve it. The repository ships the pretrained checkpoint, the gene vocabulary and a Colab-runnable fine-tuning notebook, but carries no license file, leaving the terms of reuse unstated; the preprint remains at version 1 and has not been peer reviewed.