Cell2Sentence-Scale

Whether a drug makes a tumor cell visible to the immune system can depend on what is already happening around it: a compound that raises antigen presentation in a biopsy with low-level interferon signaling may do nothing to the same cell type in a dish. A model that scores one expression vector at a time has nowhere to put that context. C2S-Scale is built so that context is simply more tokens — neighboring cells, sample metadata, and free-text annotation all sit in the same prompt as the expression profile.

The mechanism being scaled is the cell sentence introduced by Cell2Sentence: a cell's genes are sorted by descending expression and written out as a space-separated list of gene names that any language model can read. C2S-Scale leaves that transformation untouched and pushes on everything around it — a family of decoder-only models from 410 million to 27 billion parameters built on Gemma-2 and Pythia backbones, trained on a multimodal corpus of over 50 million human and mouse cells with the metadata and paper abstracts attached to them. It is a collaboration between the van Dijk lab at Yale University, Google Research and Google DeepMind, trained in JAX on Google TPU v5 hardware.

The backbone architecture is unmodified, so C2S-Scale carries no biology-specific inductive bias of the kind built into scGPT, Geneformer or scFoundation. Everything it knows about transcriptomics arrives through the data, which is what lets one checkpoint absorb text tasks — question answering, cluster captioning, abstract generation — alongside numerical ones.

#Key Features

• Multi-cell context: An 8,192-token window holds several cell sentences at once, so tissue-level questions and spatial niche prediction are posed over a group of cells, not one profile at a time.
• Unified text and expression corpus: Training samples pair cell sentences with natural-language instructions, dataset annotations and manuscript abstracts, aligning transcriptomic state with the language biologists describe it in.
• Reinforcement fine-tuning on gene programs: Group Relative Policy Optimization refines the supervised model with rewards computed over pathway-level gene sets — apoptosis for L1000, interferon response for cytokine stimulation — instead of agreement on every gene.
• Scaling behavior in biology: Performance on annotation, dataset interpretation and conditional generation improves consistently with model size and training compute across the 410M–27B range, in both full and parameter-efficient fine-tuning.

#Technical Details

The pretraining corpus assembles more than 50 million single-cell transcriptomes from CELLxGENE and the Human Cell Atlas — over 800 datasets and, per the released model cards, more than 57 million human and mouse cells — formatted into roughly 150 million multi-task training samples spanning over one billion tokens. Model sizes are 410M, 1B, 2B, 9B and 27B. Benchmarks run against expression-only foundation models, general-purpose LLMs including GPT-4o and Gemini, and baselines such as scVI; perturbation response is scored against scGen, scGPT and CellOT by MMD, Wasserstein distance and the paper's single-cell Fréchet Inception Distance (scFID). GRPO adds further gains in Kendall's τ and Pearson's r on the L1000 apoptosis and cytokine interferon tasks. HuggingFace carries the Gemma-2 2B and 27B checkpoints under CC BY-4.0 plus a 1B Pythia pretrained checkpoint; training code is Apache-2.0.

#Applications

Inference is plain text: a prompt names the gene count and organism, supplies the cell sentence, and asks for a cell type, tissue, caption, generated cell or answer. That suits atlas annotation without marker-gene curation, generation of cells conditioned on a cell type or tissue, prediction of how a profile shifts under a genetic or chemical perturbation, and in-silico screening of thousands of compounds against a baseline state. Niche labeling and neighbor generation extend the same interface to imaging-based spatial transcriptomics.

#Impact

What distinguishes C2S-Scale from its predecessor is a dual-context virtual screen. Over 4,000 compounds were simulated against two backgrounds — primary tumor profiles with intact tumor-immune interaction and low baseline interferon signaling, and cell line profiles without it — and the model was asked which would raise MHC-I antigen presentation only in the first. Sixteen of the top 23 hits were reported modulators or mechanistically plausible; the screen also flagged silmitasertib (CX-4945), a CK2 inhibitor with no literature linking it to MHC-I. In a tumor fragment model, 5 of 6 tested hits significantly raised surface MHC-I; in WAGA Merkel cell carcinoma cells, absent from training, silmitasertib alone did nothing while its combination with low-dose interferon-β gave a reproducible dose-dependent increase, reported by Google as roughly 50%. The authors present the conditional reasoning behind that prediction as a capability that emerges with scale, and note that the mechanism remains uncharacterized. The work is a preprint under a CC BY-NC-ND license, and the rank-ordering transformation still discards absolute expression magnitude — but it is among the clearest demonstrations that a general-purpose language model, given transcriptomes as text, can generate a hypothesis that survives contact with a bench.