GeneMamba

Single-cell foundation models inherited the transformer along with everything else from language modelling, including its quadratic cost in sequence length. That cost is awkward for transcriptomics, where a single cell is naturally represented as a ranked list of thousands of genes and where the useful context window keeps growing. GeneMamba replaces the transformer stack with bidirectional Mamba blocks — a selective state space architecture that scales linearly in sequence length — and asks whether a single-cell foundation model can keep transformer-level accuracy at substantially lower cost.

The model is pretrained on roughly 30 million human cells and produces cell embeddings that are used directly, without per-dataset retraining, for clustering, cell type annotation, multi-batch integration, and gene-gene correlation analysis. Two design choices distinguish it from the transformer-based cohort it competes with. Genes are encoded by expression rank rather than by binned expression values, following the convention established by Geneformer, with a median-based normalisation that deprioritises housekeeping genes. And pretraining adds a pathway-aware contrastive objective on top of the usual next-gene prediction, so that genes sharing a biological pathway are pulled together in representation space.

GeneMamba was developed at the New Jersey Institute of Technology and first posted as a preprint in April 2025. It remains a preprint; the reported benchmarks are the authors' own and have not been independently reproduced.

#Key Features

• Linear-time sequence modelling: Bidirectional Mamba blocks with RMSNorm replace self-attention, removing the quadratic scaling that constrains context length in transformer-based single-cell models such as scGPT and scBERT.
• Rank-based gene encoding: Each cell is tokenised as its genes ordered by normalised expression, with the top 2,048 or 4,096 genes taken as input over a vocabulary of 25,426 Ensembl gene identifiers.
• Pathway-aware contrastive loss: An InfoNCE term encourages genes in a shared pathway to be represented similarly, combined with the next-gene language loss at a weight of 0.1.
• Frozen-embedding downstream use: Cell embeddings are pooled from the backbone and consumed directly by clustering, annotation, and integration workflows; a sequence-classification head is available for fine-tuning.
• Hugging Face-native checkpoint: The released weights load through the standard transformers interface with trust_remote_code, alongside the gene tokenizer and worked examples for embedding extraction, fine-tuning, and pretraining from scratch.

#Technical Details

Pretraining data is drawn from CELLxGENE. About 50.7 million cells were collected, 41% removed as duplicates, and quality filtering left 29,849,897 cells spanning diverse human tissues. The published checkpoint is a 24-layer bidirectional Mamba backbone with an inner dimension of 512 and 65.7 million parameters, trained for five epochs on four NVIDIA A100-SXM4-80GB GPUs over roughly three weeks; the model card additionally describes 48-layer and 768-dimensional configurations, which are not currently published as separate repositories. Maximum input length is 2,048 tokens and cell embeddings are mean-pooled over the sequence.

On cell type annotation the model reaches 0.9713 accuracy and 0.7710 macro-F1 on hPancreas and 0.9603 accuracy with 0.9235 macro-F1 on the Myeloid benchmark. On multi-batch integration it reports an average batch-correction score of 0.9604 with an average biological-conservation score of 0.8344 on PBMC12k, and 0.9673 batch correction on perirhinal cortex data. A gene rank reconstruction task on PBMC12k gives a BLEU score of 0.987 and Spearman correlation of 0.711, ahead of Geneformer. Comparisons throughout are against scGPT, Geneformer, scFoundation, scBERT, and Harmony.

#Applications

The model targets the standard single-cell analysis stack: embedding cells for clustering and visualisation, annotating cell types in a new dataset, integrating experiments that span batches or donors, and examining gene-gene relationships learned during pretraining. Its practical draw is cost. Groups that hit memory or throughput limits pushing thousands of genes per cell through an attention-based model can run the same workloads with linear-time scaling, and the Hugging Face packaging means adoption is a from_pretrained call rather than a bespoke training environment. Labs already committed to a transformer-based pipeline get a like-for-like alternative to benchmark against on their own data.

#Impact

GeneMamba is among the clearer demonstrations that state space models transfer to single-cell transcriptomics rather than being confined to natural language and long-genomic-sequence settings, and its combination of rank encoding with a pathway-aware contrastive objective is a concrete proposal for injecting prior biology into pretraining. The caveats are real: the work has not been peer reviewed, all reported comparisons come from the authors, and the code repository ships no license file even though the Hugging Face weights repository carries Apache 2.0 terms. Pretraining data must be reassembled through the CELLxGENE API or a sample archive rather than downloaded as a released corpus, which raises the cost of exact reproduction.