PathDiff

Generative models offer a way around the data scarcity that privacy restrictions impose on computational pathology, but the two control signals pathologists most want are never available together. Diagnostic text reports carry high-level semantics — tumor type, staining, tissue context — while cell-type masks carry the fine spatial structure that downstream segmentation models actually learn from. Public histopathology datasets pair images with one or the other, never both, which forced earlier conditional generators to commit to a single modality and give up the complementary one.

PathDiff, from a team at the University at Buffalo with collaborators at the University of Florida and Harvard Medical School, removes that constraint. It learns from unpaired text-image and mask-image corpora simultaneously, projecting both condition types into one shared conditioning space during joint training. At sampling time the model can be driven by text alone, a mask alone, or an arbitrary text-plus-mask pair that appears in no training set — for example, a report describing signet ring cell carcinoma composed onto a nuclei layout drawn from a different dataset. The work was presented at ICCV 2025.

Where pathology vision-language models such as PLIP and PathAsst read slides, PathDiff writes them, targeting the augmentation of nuclei segmentation and classification training sets rather than diagnosis itself.

#Key Features

• Unified conditioning from unpaired data: A joint training scheme alternates between text-image and mask-image batches, substituting a null condition for whichever modality the batch lacks, so both controls end up in a single latent conditioning space.
• Composable, never-seen condition pairs: Because the two conditions are learned independently but represented jointly, users can compose semantic and spatial control at inference without any paired training triplets.
• Pathology-specific text grounding: Text is encoded with PLIP, a CLIP model trained on pathology image-caption data, rather than a general-domain text encoder.
• Stronger fidelity than single-condition baselines: PathDiff records the best CLIP-FID and KID against Diffmix, SDM, and ControlNet on all three mask-image benchmarks.
• Measurable downstream benefit: Adding PathDiff samples to real training data improves nuclei segmentation and classification over training on real data alone.

#Technical Details

PathDiff is a latent diffusion model built on a shared VAE encoder-decoder pretrained on TCGA-BRCA histopathology. A U-Net first pretrained on the text-to-image corpus is frozen; spatial control follows the ControlNet recipe, duplicating the U-Net's downsampling and middle blocks with zero-convolution outputs injected into the skip connections. Cell-type masks pass through a shallow CNN embedder, and PLIP text embeddings are cross-attended into the U-Net. Only the copied encoder and mask embedder are trained. Text-image supervision comes from a 100K H&E-stained subset of PathCap; mask-image supervision from PanNuke, CoNIC, and MoNuSAC. Training ran for 60 epochs on four NVIDIA A6000 GPUs at 256×256 resolution, with DDIM sampling at 200 steps and a classifier-free guidance scale of 1.75.

On the CoNIC test split PathDiff reaches a CLIP-FID of 5.54 and KID of 0.0488, against 6.79 for ControlNet, 9.80 for SDM, and 8.58 for Diffmix; it leads on MoNuSAC (6.99) and PanNuke (7.21) as well, and posts the best faithfulness scores on all three. Conditioned on text alone it attains a PLIP image-text similarity of at least 24.66 — close to the 26.56 measured on real PathCap images — versus 21.76–22.95 for ControlNet. Using the synthetic images to augment a CellViT segmentation model with a HoVer-Net decoder raises PanNuke Dice to 0.8164 and classification F1 to 0.8161.

#Applications

The primary use is data augmentation for nuclei segmentation and classification, where annotated slides are expensive and rare morphologies are underrepresented. Because conditions are composable, researchers can request specific diagnostic contexts at specific cellular layouts — a practical route to balancing long-tailed cell-type distributions or stress-testing a segmentation model on configurations absent from its training data. Synthetic slides also sidestep the sharing restrictions attached to real patient tissue, though the released checkpoint is distributed for research and educational purposes only and is not intended for clinical use.

#Impact

PathDiff establishes that unpaired supervision is sufficient to build a multi-conditional generator, a useful result for medical imaging generally, where paired annotation across modalities is the exception. Code is public on GitHub and a pretrained checkpoint is distributed via Google Drive, but the repository carries no open-source license file, and hosting weights outside a versioned model hub makes long-term availability less certain. Evaluation of the unified text-plus-mask setting also relies on silver-standard masks generated by an auxiliary model, since no public dataset supplies genuine text-mask-image triplets.