Loading…
Histopathology image translation with diffusion, moving H&E tiles between stains, tumor types, and organ sites and editing them from omics profiles.
No providers recorded yet. Browse all providers
A pathology lab that wants to convert cryosectioned slides to FFPE-like images, virtually stain H&E for a dozen antibodies, and restyle tiles between tumor types has historically needed a separate generative adversarial network for each source-target pair — covering n domains pairwise takes n(n-1)/2 models. Worse, the best translators earn their accuracy with task-specific machinery: a leading virtual-IHC model bakes a staining-consistency network into its discriminator, which is exactly what stops it being reused for anything that isn't virtual staining.
His-MMDM (Histopathological image Multi-domain Multi-omics translation with Diffusion Models) replaces the pair-of-domains framing with a conditional one. A single denoising diffusion model is trained over all the domains at once, and translation runs the forward diffusion conditioned on the source domain and the backward denoising conditioned on the target — the dual diffusion implicit bridges construction, which composes two score-based models into a mapping between their data distributions. Because the condition is an embedding added into every U-Net layer rather than an architecture choice, the same model accepts categorical labels, per-gene somatic mutation status, and quantitative transcript levels alike. The omics conditions are what make it an editor rather than a synthesizer: it takes a real tile and asks what the tissue would look like if TP53 were mutated or the mitotic spindle pathway were upregulated.
The model came from Xin Gao's group at KAUST with clinical collaborators at Harbin Medical University, and was published in Advanced Science after a 2024 medRxiv preprint.
His-MMDM is a classifier-guided denoising diffusion probabilistic model with a U-Net noise predictor operating on 128×128 tiles, sampled with DDIM as the ODE solver. Genomic conditioning uses paired wild-type and mutant embeddings per gene summed through an MLP; transcriptomic conditioning multiplies a gene embedding by an expression-level network. The genomic head spans 522 genes (334 from ten oncogenic signaling pathways plus 188 high-frequency or discriminative genes) and the transcriptomic head 4,461 genes (4,193 from MSigDB's 50 hallmark pathways plus 268 more). Training used 24 NVIDIA V100 GPUs over three cohorts: 22,596 TCGA whole-slide images across 19 tumor types, 475 primary and metastatic lung tumor slides from 400 patients at Harbin Medical University Cancer Hospital, and 6,200 glioma and meningioma slides from 557 patients at its First Affiliated Hospital, the last covering H&E plus 14 IHC markers, the 13 common in brain tumor diagnosis being the ones the virtual-staining model is trained on. On cryosection-to-FFPE conversion, measured by the reduction in Fréchet Inception Distance, the general model matched the dedicated AI-FFPE and beat it in 7 of 19 tumor types, rising to 18 of 19 against AI-FFPE with its application-specific components disabled. Translated cryosections raised image retrieval accuracy by 0.30 for PLIP and 0.27 for CHIEF, and PLIP's text-guided classification accuracy by 0.39.
The immediate clinical use is intraoperative: frozen sections read poorly for pathologists and worse for FFPE-trained encoders, and translating them recovers both. Virtual IHC staining gives a first-pass read on marker panels without consuming tissue or antibody. Omics-guided editing is aimed at teaching rather than diagnosis — paired real-and-edited tiles served as training material and improved pathologists' accuracy at recognizing the underlying alterations. Tumor-type translation supports transfer to cancers with too few labeled slides for their own classifier, and the compatibility score of a primary tumor tile translated to a metastatic site tracks TNM staging and overall survival.
His-MMDM shows that associations between molecular state and tissue morphology, previously established only by discriminative models predicting mutations from images, can be run in the generative direction and inspected visually. Code and pretrained checkpoints are MIT-licensed and the study cohorts are deposited on Zenodo, though the checkpoints are served from a plain S3 bucket without versioning. Two constraints bound its use: sampling a batch of tiles takes minutes on eight V100s, and each checkpoint works at a single magnification and resolution over the domain vocabulary it was trained on, so a new domain needs a new model rather than a new condition.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.