PertFormer

Predicting how a cell responds to a perturbation — silencing an enhancer, knocking out a transcription factor, or forcing a fate transition — normally costs a dedicated CRISPR screen, and those screens exist for only a handful of genes in common cell types. Computational alternatives split into two camps that each miss part of the problem. DNA sequence models such as Enformer, Borzoi, and EpiGePT predict expression and epigenomic signal from long sequence context but can only perturb the sequence itself, not chromatin state or a targeted gene. Transcriptomic foundation models such as scGPT, Geneformer, and scFoundation operate on expression vectors in which a perturbation changes fewer than 0.1% of genes, and generally require fine-tuning on labeled perturbation data first.

PertFormer (Perturbation Transformer) is a 3-billion-parameter multimodal foundation model built to close that gap, developed at DAMO Academy and Hupan Lab with collaborators at Westlake University, Guangzhou National Laboratory, and Harvard Medical School. It was posted as a preprint in December 2024 and substantially revised in September 2025. Rather than perturbing a transcriptome, PertFormer perturbs the regulatory input: an in silico knockout erases chromatin accessibility signal at motif-matched genomic loci, an in silico knock-in inserts it, and the model reads out the resulting genome-wide change in gene expression. Because the pretraining objective already maps DNA plus accessibility to expression, no task-specific fine-tuning is needed — every result reported in the paper is zero-shot.

#Key Features

• Zero-shot perturbation simulation: Knockout and knock-in effects are predicted without any labeled perturbation data, by editing accessibility signal at transcription-factor motif loci and re-running the forward pass.
• Two-stage multimodal architecture: A short-range module extracts features from 1,024-bp genomic elements, and a long-range module integrates those elements across the 300 kb flanking each transcription start site.
• Interpretable attention readouts: Per-element Attention Scores identify functional enhancers and repressors, and per-gene Attention Features cluster single cells and expose cell-type-specific master regulators.
• Bulk and single-cell pretraining: Nine multiomic modalities are combined, inheriting histone-mark and CTCF signal that single-cell assays cannot measure while retaining cellular heterogeneity.
• Open code and weights: The implementation and the pretrained checkpoints are both released under Apache 2.0, alongside a hosted web server for users without GPU access.

#Technical Details

PertFormer comprises two modules. PertFormer-Elementary embeds DNA sequence together with any combination of multiomic signals, applies a transformer over 128-bp bins, then a second transformer pooling bin-level [CLS] representations into one genomic-element embedding. PertFormer-Regulatory then models one gene at a time, drawing multiple 1,024-bp elements from within 300,000 bp of the transcription start site, combining them by attention pooling, and encoding position through a TSS-distance embedding in place of a conventional positional embedding; cross-attention captures long-range interactions among elements. Pretraining used two purpose-built corpora: bulkMultiomeCorpus-55b, over 55 billion bp of ENCODE H3K4me3, H3K4me1, H3K27ac, H3K27me3, H3K9ac, DNase, and CTCF profiles across 34 human cell lines and tissues; and scMultiomeCorpus-1.5b, 1.5 billion paired scRNA/scATAC associations from 1 million cells profiled on 10x Genomics multiome. Elementary was trained self-supervised on the bulk corpus, then frozen while Regulatory learned to predict expression from accessibility.

On K562 CRISPRi-FlowFISH enhancer-gene pairs, Attention Score ranking improved auPRC by an average of 60.3% over the best prior methods, and by 116.4% on VISTA enhancers. In silico knockout of validated enhancers and repressors reproduced the experimental direction of change with a macro-F1 of 0.759, and transcription-factor knockout and knock-in reached macro-F1 of 0.789 and 0.754 — above fine-tuned baselines by 14.8–29.9%, while zero-shot baselines scored below 0.5. Simulated OSKM induction in fibroblasts, cardiomyocytes, and B cells drove predicted profiles toward human embryonic stem cells (macro-F1 0.795).

#Applications

PertFormer is aimed at target prioritization in settings where perturbation data does not exist: disease microenvironments, multi-genic perturbations, and cell state transitions. The authors used it to nominate THAP2 in triple-negative breast cancer and ZIK1 in ovarian cancer as previously uncharacterized tumor-promoting targets, then validated both with siRNA knockdown in MDA-MB-231 and SKOV-3 cells, observing reduced proliferation and migration and RNA-seq profiles shifted toward normal epithelial controls. The same attention machinery infers gene regulatory networks and master regulators in cell types absent from pretraining, using only DNA sequence and scATAC-seq.

#Impact

PertFormer is one of the few perturbation models to pair a genome-scale regulatory backbone with wet-lab validation of its own predictions, and its Apache-2.0 code and checkpoints make that claim checkable. Caveats remain: the work is a preprint that has not been peer reviewed, benchmark comparisons are author-reported, inference is documented against NVIDIA A100 80GB hardware, the checkpoints are Python pickle files, and Alibaba has filed patent applications covering the pretraining and zero-shot prediction methods. The perturbation formulation itself — editing chromatin accessibility rather than expression — is the transferable idea, giving regulatory-genomics models a route into a problem transcriptome-only models have dominated.