Loading…
Single-cell RNA-seq foundation model pretrained only on malignant cells, for zero-shot batch integration and drug response prediction in tumors.
No providers recorded yet. Browse all providers
A tumor biopsy is a mixture, and the transcriptional programs that matter for therapy live in its malignant fraction alone. Most single-cell foundation models are pretrained on atlases dominated by healthy tissue and peripheral blood, so most of their pretraining signal describes cells a cancer biologist is not asking about. Copy-number alterations show the cost: pervasive in malignant cells, tied to progression and treatment evasion, near-absent from normal ones, they barely register in a mostly non-malignant corpus.
CancerFoundation, from Valentina Boeva's group at ETH Zurich, inverts that trade — pretrain on malignant cells only, and keep the model small. It was trained on one million high-quality malignant cells from the Curated Cancer Cell Atlas, spanning roughly 1,500 tumors across 112 studies and 46 cancer types, with InferCNV used to call each cell's copy-number status so the malignant fraction could be isolated cleanly. The encoder holds 10.8 million parameters — about ten times smaller than scFoundation, five times smaller than scGPT, some sixty times smaller than UCE. A million cells from a public atlas is a badly skewed sample, and the design choices below correct that skew rather than scale past it. The work was posted as a preprint in November 2024 and presented at the AI for New Drug Modalities workshop at NeurIPS 2024.
The encoder is a six-layer self-attention transformer with eight heads, an embedding dimension
of 256 and a hidden dimension of 512, implemented without positional encoding. Its vocabulary
holds 28,725 gene tokens — roughly 29,000 highly variable genes chosen per tissue type, against
the roughly 67,000 scGPT carries — and each cell is a fixed 1,200-token sequence. Expression is
discretized into 51 bins, then summed into the gene-token embedding through a two-layer MLP.
Pretraining is masked expression prediction at 25%, 50% or 75% masking, with the attention mask
modified so unmasked tokens cannot attend to masked ones, plus a cell-level objective on a
<cls> token. Training ran 15 epochs across eight GPUs with Flash Attention and mixed
precision.
On zero-shot batch integration of the Neftel glioblastoma cohort — a dataset the model never saw — it scores an aggregate 0.58 against 0.54 for both scGPT variants and 0.47 for scVI, and separates all four malignant states where the baselines blur MES-like into AC-like. For drug response, a DeepCDR-style head over frozen embeddings predicts IC50 on CCLE and GDSC with both cell lines and drugs held out; Pearson's R for lapatinib rises from 0.08 under scFoundation embeddings to 0.95. The code and checkpoint call the model CancerGPT internally, a name shared with an unrelated LLM-based drug-synergy predictor.
The natural use is as a drop-in encoder for cancer scRNA-seq: embed a newly sequenced tumor cohort, integrate it across patients and platforms without training anything, then read transcriptional states off the result. Because the embeddings also apply to bulk profiles, the same checkpoint feeds pharmacogenomic screens — ranking IC50 for drug–cell-line pairs neither seen in training — and patient-stratification models built on TCGA-scale expression data, all from a conda environment and a repository clone.
CancerFoundation is a concrete argument that domain focus can substitute for scale in single-cell modeling, matching or beating encoders trained on 30–50 million mostly non-malignant cells while using a fiftieth of the data. The authors are candid about where that stops: on their proposed survival-prediction benchmark over 21 TCGA cancer types, single-cell foundation models including this one largely fail to beat a Cox model fitted on raw gene expression, and CancerFoundation trails that baseline for cancer types thinly represented in pretraining. The work remains a preprint and a workshop paper, and the weights ship from an institutional file share rather than a versioned registry. It has nonetheless been adopted as a fixed encoder downstream, by STMDiT among others.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.