scKITE

Single-cell foundation models have largely been scaled by adding transcriptomes: Geneformer on roughly 30 million profiles, scGPT on 33 million, scFoundation on 50 million. scKITE's own ablation marks where that strategy runs out: the same architecture trained on expression alone reached a downstream performance plateau at half of its pretraining corpus, and the remaining half bought almost nothing. The useful question then is not how many more cells to read, but what else the pretraining objective could be told about each one.

scKITE — single-cell Knowledge-Integrated Transformer, from China Agricultural University — answers by hanging two lightweight autoregressive decoders off a shared transcriptomic encoder during pretraining. One must regenerate the cell's natural-language description: its identity, tissue of origin, disease or physiological condition, donor characteristics. The other must emit the transcription-factor regulons active in that profile, each TF followed by its importance-ranked target genes. Both reach the encoder only through cross-attention, so the only way to satisfy them is for its hidden states to carry cell identity and regulatory structure. Neither decoder survives: both are discarded after pretraining, and downstream work uses the encoder alone on transcriptomic input, requiring no annotation or regulon information at inference.

The result is a data-efficiency argument. Pretrained on 179,067 pseudo-bulk profiles — under 0.5% of what the models above consumed — the frozen encoder was evaluated on five cell-type annotation atlases, four batch-integration datasets and two Perturb-seq screens, none of them drawn from its pretraining corpus. Averaged across four pretraining fractions, knowledge-enhanced pretraining gained 28.8% relative to the identical architecture trained on expression alone.

#Key Features

• Knowledge decoders used only during pretraining: The annotation and regulon decoders supply supervision and are then removed, leaving a general-purpose encoder rather than a multi-task model that needs side inputs at inference.
• Zero-shot cell-type annotation: Frozen <cls> embeddings feed a five-nearest- neighbour classifier with no trained head, reaching a mean macro-F1 of 0.500 across five benchmarks against 0.430 for scGPT and 0.399 for Geneformer.
• Batch integration without adversarial correction: With all parameters frozen, mean AvgBIO rose from 0.338 to 0.474 and mean AvgBATCH from 0.836 to 0.938 over the knowledge-free ablation across four datasets.
• Contextual gene embeddings for perturbation: Gene-level representations substitute for the native embeddings in GEARS, lowering Top-20 differentially expressed gene MSE on both the Norman and Replogle screens.
• Attention that can be audited: Cross-attention ranks marker genes above expression-matched background in 196 of 198 cell types, and CollecTRI-supported TF targets above matched non-targets for 32 of 45 transcription factors.

#Technical Details

The backbone is 12 pre-normalisation Transformer encoder blocks, hidden dimension 512, eight attention heads and dropout 0.1. Each profile keeps up to 2,048 expressed genes ordered by descending expression, prefixed with a <cls> token for a maximum sequence length of 2,049; positive values are quantile-binned within each profile into 50 non-zero levels. The two knowledge decoders are two blocks each, same width, independent parameters. Pretraining runs in two stages on one corpus: Stage 1 is masked-expression reconstruction at 30% masking, and Stage 2 initialises from that checkpoint, drops masking to 10% and adds the two autoregressive cross-entropy objectives at equal weight.

The corpus is the CELLxGENE component of CellWhisperer — 376,983 human pseudo-bulk transcriptome–annotation pairs, split into a 358,134-profile training pool and a fixed 18,849-profile validation set. The released configuration uses the 179,067-profile nested geometric sketch of that pool. Regulons come from pySCENIC, with GRNBoost2 co-expression inference refined by cisTarget motif enrichment and scored by AUCell, leaving 530 informative regulons; up to three are serialised per profile per epoch. After full fine-tuning, mean macro-F1 across the five annotation benchmarks reaches 0.668 against 0.561 for scGPT and 0.543 for Geneformer. Cross-tissue cell-type retrieval accuracy in the Cross-tissue Immune Cell Atlas rises from 42.8% to 66.2%, and recovered genetic-interaction magnitude for held-out double perturbations correlates at Pearson r = 0.337 against 0.229 for stock GEARS.

#Applications

The frozen encoder suits groups annotating new scRNA-seq datasets where labelled reference data are thin, integrating cohorts split across assays, donors or studies, and prioritising genetic perturbations before running them. Because inference needs only an expression profile, the model slots into existing pipelines without the text or regulatory inputs its pretraining consumed.

#Impact

scKITE is a preprint, and neither code nor weights have been released, so its results are not yet independently reproducible. Its contribution is a claim about where scaling effort should go: that pairing transcriptomes with cell-level descriptions and gene-level regulatory programs shifts the data-scaling curve rather than merely climbing it. The authors bound that claim carefully. The supervision is only as good as its sources — annotations derived from dataset metadata vary in completeness and specificity, and pySCENIC regulons are predicted TF–target relationships, not experimentally established causal networks. The data-efficiency pattern was measured within one combination of corpus, architecture and benchmark suite, and is not offered as a universal threshold.