scLong

Most single-cell foundation models never look at most of a cell's genes. Geneformer reads 2,048 genes per cell and scGPT about 2,000, in both cases the highest-expressed ones, so transcription factors, receptors and other regulators sitting at a handful of counts are discarded before the model reads anything. Those low-expression genes are frequently the switches that modulate the abundant genes the model does get to see, so dropping them leaves an incomplete picture of regulation.

scLong removes the selection step. It runs self-attention across all 27,874 human genes in its vocabulary, roughly 20,000 protein-coding and 8,000 non-coding, so a cell is represented by its whole measured transcriptome, zeros included. A second departure is that gene identity is not inferred from expression alone: every gene token carries an embedding from a graph convolutional network run over a Gene Ontology similarity graph, supplying a functional prior for exactly the genes that expression data says least about.

The model was developed at UC San Diego and MBZUAI, with collaborators at Carnegie Mellon and Scripps Research, and published in Nature Communications in 2026.

#Key Features

• Whole-transcriptome self-attention: All 27,874 genes in the vocabulary take part in every forward pass, about 20,000 protein-coding and 8,000 non-coding, so no gene panel is chosen up front.
• Gene Ontology priors through a GCN: Each gene is joined to the 20 others whose GO annotation sets overlap it most, and a one-layer graph convolution over Gene2vec vectors turns that graph into gene embeddings.
• Split attention budget: A large Performer encoder handles the 4,096 highest-expressed elements of a cell, a much smaller one handles the remainder, and a full-length Performer then attends across all of them.
• Zero-shot batch integration: The pretrained representations align cells collected on different sequencing platforms without any tuning on the target dataset.
• Task decoders for four prediction problems: Genetic and chemical perturbation response, cancer drug response including drug pairs, and gene regulatory network inference each attach a decoder to the pretrained gene representations.

#Technical Details

scLong has one billion parameters, roughly ten times scFoundation and GeneCompass at 100 million each. Each element of a cell's expression vector is the sum of a gene embedding and an expression embedding: the gene encoder is the Gene Ontology graph convolution over Gene2vec initialisations, the expression encoder a two-layer perceptron, both 200-dimensional. Elements are ranked by expression and split — the top 4,096 go to a 42-layer Performer with 32 attention heads and hidden dimension 1,280, the rest to a two-layer Performer with hidden dimension 200 — before a final two-layer full-length Performer attends across all 27,874. Performer's linear attention is what makes a sequence of that length affordable.

Pretraining masks 15% of the non-zero values in each cell and minimises squared error on the reconstructions, across 48,024,242 cells drawn from CELLxGENE, Cell BLAST and the Human Cell Atlas — 1,618 datasets spanning more than 50 tissues. Five epochs took about 35 GPU-days on 96 A100s in bfloat16. On the Norman perturbation benchmark with a GEARS decoder, scLong reaches Pearson 0.63 on unseen single-gene perturbations against 0.56 to 0.58 for GEARS, Geneformer, scGPT and scFoundation. Cancer drug response on the DeepCDR split gives Pearson 0.873 against 0.852 for Geneformer; regulatory network inference on human embryonic stem cells gives an AUPR ratio of 1.35 against 1.12; zero-shot batch integration on a six-batch pancreas dataset gives a batch ASW of 0.96, ahead of scGPT at 0.89 and UCE at 0.83. Ablations that remove the low-expression genes or randomise the Gene Ontology graph degrade every one of those results.

#Applications

scLong suits translational single-cell groups forecasting how an expression profile shifts after a CRISPR knockout or a compound exposure, ranking anti-cancer drugs and drug pairs for a given cell line, and reconstructing regulatory networks for one specific cell type rather than reading them off a generic interaction database. Because batch integration works zero-shot, the embeddings are usable directly for harmonising datasets collected on different platforms. Full-transcriptome coverage matters most where the biology runs through lowly expressed regulators or rare cell populations, precisely where a preselected panel is likeliest to omit the signal.

#Impact

scLong demonstrates that attending over an entire transcriptome is affordable and that it pays off: ablations dropping the low-expression genes or randomising the Gene Ontology graph lose most of the gains, so neither design choice is decoration. The costs are real: inference is heavy for groups without cluster access, and the Gene Ontology graph is a static curated resource that cannot express condition-specific regulation. Performance also depends on where the high/low expression split is drawn, which may not suit cell types with unusual expression distributions. Code, pretrained weights and the preprocessed pretraining corpus are all publicly downloadable, though the repository carries no license file.