Tabula

Single-cell foundation models have converged on a convention borrowed from language: treat a cell as a sentence and its genes as words, then apply masked or autoregressive modeling. The convention is awkward, because scRNA-seq measurements have no natural order — a cell is a row in a cell-by-gene table, and any sequence imposed on its genes is arbitrary. It is also expensive in a second sense: pooling ever-larger corpora into one central repository raises privacy and consent problems for clinical and proprietary data that will only sharpen as these corpora grow.

Tabula addresses both at once. Developed by Jiayuan Ding, Xiaojie Qiu, and colleagues at Stanford University with Michigan State University, Central South University, and Johns Hopkins University, it replaces the sentence framing with tabular modeling — each gene is a column token, and self-supervised objectives operate along both the cell axis and the gene axis — and replaces centralized pretraining with federated learning, in which eight tissue-specific clients train locally and share only model weights with a central server.

The result is a deliberately small model that competes with much larger ones. Pretrained on 15 million cells, roughly half what scGPT used and under a third of scFoundation's corpus, Tabula reports better cell type annotation than scGPT, Geneformer, and scFoundation on all four standard metrics for the human pancreas benchmark.

#Key Features

• Tabular pretraining objective: Self-supervised losses run cell-wise across rows and gene-wise down columns, so the model learns column dependencies rather than a spurious gene ordering.
• Federated training across tissues: Eight clients — intestine, pancreas, lung, heart, blood, kidney, brain, and a catch-all — train locally, with the shared tabular transformer averaged across clients at the end of every epoch.
• Tissue-specific embedders: Each client keeps its own embedder and projection head while the transformer is shared, letting heterogeneous tissue distributions coexist in one model.
• Zero-shot regulatory inference: Using the same pretrained weights and no fine-tuning, in-silico perturbation predicts pairwise regulation with accuracy above 0.83 across hematopoiesis, pancreatic endogenesis, neurogenesis, and cardiogenesis, and extends to three-gene motifs such as GATA1, FLI1, and KLF1.
• Cell- and gene-level task coverage: One backbone supports cell type annotation, batch integration, and multi-omics integration at the cell level, and imputation, GRN inference, and perturbation response at the gene level.

#Technical Details

The architecture is compact: three transformer blocks, eight attention heads each, and a 192-dimensional output embedding, accelerated with FlashAttention-2. Input is capped at 1,200 highly variable genes per dataset, and a [CLS] token aggregates column embeddings into the cell representation. Pretraining drew 15 million cells from the CELLxGENE census, sampled from a 37.99-million-cell human collection grouped into eight tissue categories and capped at 3 million cells per category to keep the clients balanced. Training used AdamW, with client weights taken from epoch 8 as the final tissue-specific models; clients ran on separate compute nodes communicating over InfiniBand via NCCL.

An ablation across the four combinations of {federated, centralized} × {tabular, masked language modeling} on a 1-million-cell subset found tabular learning ahead of MLM in both training regimes, and federated learning ahead of centralized training on cell type annotation. Against published baselines, Tabula led scGPT on accuracy, precision, recall, and macro-F1 for hPancreas annotation, with Geneformer and scFoundation further behind, and outperformed scGPT and Geneformer on Pearson-delta for perturbation response on the Adamson and Norman datasets. A scaling analysis showed annotation accuracy improving as the pretraining corpus grew from 300,000 to 3 million to 15 million cells.

#Applications

Tabula targets settings where the data cannot be centralized: hospital consortia, biobanks, and companies holding proprietary single-cell atlases that want a shared model without sharing cells. Downstream, it serves the standard analysis stack — annotating cell types in a new dataset, imputing dropouts, correcting batch effects, aligning modalities, and screening candidate perturbations before running them — and its zero-shot regulatory inference offers a way to nominate combinatorial regulators in developmental systems where exhaustive experimental screens are impractical.

#Impact

The entry Tabula makes into the single-cell foundation model literature is less about a headline benchmark than about two structural arguments: that the tabular geometry of expression data deserves a matching inductive bias, and that federated training is not a performance sacrifice. Both are supported by controlled ablations rather than asserted. The work remains a preprint awaiting peer review. Code, tutorials for five downstream tasks, and pretrained checkpoints are public, with weights distributed through a linked Google Drive folder and loaded via the repository's get_pretrained_model helper, but the repository carries a non-commercial license that bars commercial use. The federated design also has a limitation the authors name: all clients share one general transformer while using distinct embedders, so a universal embedder remains future work.