CoSyne Perturb-seq scFM

Genome-scale Perturb-seq atlases knock down thousands of genes and read out the transcriptome of every cell, but the per-gene signal is faint. Standard analysis calls differentially expressed genes for each perturbation, and for most perturbations it calls none — so most of the data is discarded before any downstream analysis. The failure mode is structural: differential expression looks for large shifts in individual genes and is blind to a perturbation that nudges thousands of genes a little.

This model, from CoSyne Therapeutics in London, reframes the problem as representation learning. The team pretrained a single-cell foundation model in the mould of Geneformer, then fine-tuned it with a contrastive objective on Perturb-seq data: embeddings of two unperturbed cells are pulled together, and embeddings of a perturbed and an unperturbed cell are pushed apart. The result is an embedding space in which the distance between a perturbation's cells and control cells is itself the measure of effect size, computed transcriptome-wide rather than gene by gene. The paper does not assign the model a name, so it is catalogued here under a descriptive one.

The work was published as a conference paper at ICLR 2025 and posted to bioRxiv in April 2025. It is one of two single-cell models CoSyne released that month, alongside Bio-DTA, which links DNA-sequence and transcriptome representations.

#Key Features

• Perturbation-aware embeddings: A single forward pass yields a cell embedding whose distance from control embeddings quantifies the perturbation's global transcriptomic effect, with no per-perturbation model fitting.
• Generalizes to unseen targets: Perturbation targets held out of fine-tuning, including the pairs LARP7 versus MRPS33 and PCBP2 versus POLRMT, separate in the fine-tuned embedding space even though the model was never trained to discriminate between perturbations.
• Recovers signal where differential expression finds none: Applied to 6,248 perturbations with zero called differentially expressed genes, the highest-distance perturbations are significantly enriched for cell-cycle and essential genes.
• Contrastive fine-tuning as a general recipe: The objective is a standard margin-based pair loss backpropagated through the whole encoder, applicable to any masked-language-model single-cell backbone.

#Technical Details

The backbone is a bidirectional encoder-only transformer that consumes each cell as a sequence of up to 2,048 gene names ordered by median-normalized expression. It was pretrained with a masked language modelling objective, masking 15% of tokens for three epochs, on roughly 33 million cells drawn from 265 datasets in the CellXGene census (version 2023-07-25), excluding cancer cells and cells with fewer than 500 expressed genes. Cell embeddings are the mean of the penultimate layer's gene embeddings, a 256-dimensional vector. On a dataset published after both models' training cutoffs, the backbone performed comparably to Geneformer and slightly better at reproducing the ranking of highly expressed genes.

Fine-tuning used the Replogle et al. 2022 atlas of 9,866 knockdowns across 1.98 million K562 cells, restricted to the 1,541 perturbations with at least 20 differentially expressed genes, split 80/10/10 by target gene. Training ran for 10 epochs with a margin of 20, a learning rate of 1e-5 and AdamW, taking seven days on a single NVIDIA T4. On a test set of 232 perturbations, the fine-tuned model had the highest median separation of perturbed from unperturbed cells across energy distance, cosine energy distance, and Wasserstein distance — a median E-distance of 84.14 against 49.4 for PCA, the strongest simple baseline. The comparison is not one-sided: a single-layer scVI variational autoencoder achieved larger median E-distances between pairs of different perturbations (137.9 against 102.0), and the pretrained backbone before fine-tuning performed poorly on every metric.

#Applications

The immediate use is rescuing perturbations that conventional analysis throws away. In target discovery, a knockdown that produces a large embedding shift but no significant single-gene change is a candidate the standard pipeline would silently drop, and ranking perturbations by embedding distance surfaces them for follow-up. The same embeddings support grouping perturbations by transcriptomic phenotype, which is the basis for inferring shared pathway membership from a perturbation screen. The approach suits pharmaceutical and academic groups running genome-scale CRISPR screens with single-cell readout, where the analysis bottleneck is deciding which of thousands of hits deserve experimental attention.

#Impact

The result is a useful counterweight to benchmarks showing that single-cell foundation models are beaten by PCA on perturbation tasks: the pretrained backbone alone was indeed weak here, and only task-matched contrastive fine-tuning made it competitive. That framing — the objective, not the scale, is what was missing — is the paper's most transferable contribution. Two limits bound its reach. Fine-tuning and evaluation are confined to one cell line, K562, from a single Perturb-seq atlas, so generalization across cell types is untested; and the enrichment analysis for zero-DEG perturbations is an indirect validation, since no ground truth exists for whether those cells truly shifted. No code, weights, or trained checkpoint accompany the paper.