SATScG

A transformer trained on single cells has to decide what a token is. Geneformer ranks genes by expression and feeds the ordering; Cell2Sentence-Scale writes each cell out as a sentence of gene names; most other architectures skip discretization and embed continuous counts. SATScG learns the vocabulary instead. A residual vector-quantized autoencoder compresses a cell's 18,080-gene log-normalized profile into one 256-dimensional latent, then encodes that latent coarse-to-fine across eight codebooks of 256 entries each, the first taking the codeword nearest the latent and each later one the codeword nearest the running residual. Every cell becomes an ordered tuple of eight integers — a fixed-length block of discrete symbols an ordinary causal language model consumes unchanged.

Built by Aleksandr Sharipov, Yusif Mukhtarov and Igor Molybog at the University of Hawaiʻi at Mānoa and posted to arXiv in August 2026, the model trains on a deliberately narrow task: given expression vectors from one cell type, generate more of that type. Cells sharing an annotation are concatenated into a "cell sentence" of 32 cells separated by learned [SEP] tokens, and a LLaMA-architecture decoder is trained from scratch with next-token cross-entropy over the codes. The label is never shown to the network — it only governs which cells may share a sequence — so at inference the model is prompted with held-out measured cells of a target type and continues the sequence, with no weight update and no per-dataset refitting.

The headline result concerns scale, not a leaderboard position. Single-cell foundation models have been reported to plateau well below the corpus sizes available, read as evidence that data diversity rather than scale is the binding constraint. SATScG sweeps model size and data size independently on a fixed architecture, fits a joint two-exponent law to the pretraining loss, and argues those plateaus are a capacity bottleneck, not a missing data-scaling law.

#Key Features

• Learned discrete tokenizer: The residual-VQ autoencoder is trained once, then frozen and used purely as a tokenizer, so every transformer in the sweep sees one vocabulary.
• In-context conditioning: Generation is prompted with real cells of the target type rather than fine-tuned per cell type, so one pretrained checkpoint serves every type.
• Jointly fit scaling law: The Chinchilla two-term form over 22 grid points gives L(N,D) = 2.90 + 2.05 × 10⁴/N⁰·⁸¹ + 4.28 × 10³/D⁰·⁵⁷, R² 0.977, both exponents clearly positive.
• Loss that tracks biology: Across ten checkpoints, mean-expression correlation rises as validation loss falls, at a Pearson r of −0.99, so the objective predicts generation fidelity.
• Modest compute footprint: Every experiment ran on a single NVIDIA H100 NVL, putting the scaling study within reach of small groups.

#Technical Details

All data is aligned to a fixed 18,080-gene panel, normalized to 20,000 counts per cell and log1p-transformed. The RQ-VAE pairs a single-hidden-layer MLP encoder of width 1,024 with a matching decoder, trained on reconstruction MSE plus a per-codebook commitment loss. The transformer holds width and depth in fixed proportion across five configurations spanning 1.31M to 83.9M parameters, adding a learned positional embedding over the eight within-cell tokens and a bias separating cell tokens from separators. A 32-cell sequence is 287 tokens.

Pretraining draws on a roughly 200-million-cell subset of Arc Institute's scBaseCount atlas (23,846 h5ad files after gene alignment and filtering), keeping cell types with at least 1,000 cells and reserving 20% of each for validation; sentence pools are sampled proportionally to cell-type abundance, avoiding the memorization uniform weighting produces on the long tail. The grid varies data budget from 38.1M to 362M cells, with held-out runs to 1.01B. Generation quality is reported for the 4.43M-parameter model trained on 121M cells, over 131 cell types: mean-expression Pearson 0.960 against 0.964 for a baseline of real held-out replicates, MAE 0.041 against 0.038, discriminability 0.990 against 0.998.

#Applications

The immediate use is synthetic single-cell data. Because generated populations sit close to empirical replicates on profile agreement and are nearly as cell-type-discriminable as real cells, the model can expand scarce or costly measurements, or supply in-silico populations of a named cell type for method development. Mean-pooling the transformer's hidden states also yields a fixed-size cell-line embedding independent of context length, an alternative to the categorical and single-profile representations used in perturbation modeling.

#Impact

SATScG enters an open argument about whether single-cell foundation models scale, where groups including the X-Cell authors have reached divergent conclusions; its controlled grid yields positive exponents on both axes and a law predicting held-out runs to within 0.5%. That claim is narrow by design, covering the pretraining loss for one architecture family rather than downstream tasks, with a parameter axis two orders of magnitude below billion-parameter models such as Tahoe-x1. The work remains a preprint, and while the reproduction code is public, no pretrained checkpoints have been released. Perturbation-response prediction is a planned second stage, not a demonstrated capability.