DiffuGTS

Tumor segmentation models are usually trained one anatomy at a time. A network that segments kidney tumors will not find a pancreatic lesion, and a model tuned on CT typically collapses on MRI. Building a separate annotated dataset for every tumor type and modality is the bottleneck, because rare cancers and rare imaging protocols are precisely where labeled volumes are scarcest. Generalizable tumor segmentation asks for one model that can find a lesion it was never trained on.

DiffuGTS, presented at CVPR 2025 by researchers at Shanghai AI Laboratory with collaborators at Zhejiang University and the University of British Columbia, approaches this by mining a generative model rather than a discriminative one. Its observation is that a medical foundation diffusion model trained to synthesize anatomically plausible CT already encodes rich, organ-specific structural knowledge, and that this knowledge can be read out for segmentation without touching the diffusion weights. DiffuGTS builds anomaly-aware open-vocabulary attention (AOVA) maps by aligning the internal visual features of a frozen diffusion backbone with text embeddings of normal and abnormal tissue descriptions, so the set of findable categories is defined by language at inference time rather than by a fixed training label list.

The second half of the method is what distinguishes DiffuGTS from earlier text-driven approaches such as ZePT and Malenia. Rather than accepting the coarse masks that vision-language alignment produces, DiffuGTS uses the frozen diffusion model to inpaint each suspected lesion into a pseudo-healthy version of the same organ, then derives a refined mask from the pixel-level and feature-level residual between the original and the healthy reconstruction. That refinement step requires no additional training.

#Key Features

• Open-vocabulary lesion queries: Text prompts describing normal and abnormal tissue drive the attention maps, so the model is not restricted to a predefined training category list and can be asked about tumor types it never saw.
• Frozen generative backbone: The medical foundation diffusion model MAISI supplies both the 3D VAE feature extractor and the generative prior; its parameters are never updated, which keeps the trainable footprint at 285M parameters versus 733–783M for comparable encoder-decoder methods.
• Training-free pseudo-healthy inpainting: The conditional generation process is reformulated as latent-space inpainting with the anomaly mask as the known region, converting pathological tissue into a healthy counterpart without any fine-tuning of the diffusion model.
• Residual-based mask refinement: Segmentation is recovered from pixel-level and feature-level differences between the original volume and its pseudo-healthy reconstruction, which sharpens boundaries that language supervision alone leaves fuzzy.
• Cross-modality transfer: A model trained only on CT tumors is applied directly to MRI, including brain gliomas, without modality-specific adaptation.

#Technical Details

DiffuGTS uses MAISI, a medical latent diffusion model capable of conditional generation from segmentation masks of 127 anatomical structures, as its frozen backbone. Text embeddings come from a frozen text encoder, and the AOVA maps are produced by CLIP-style contrastive alignment between those embeddings and MAISI's VAE features. Only the alignment and segmentation components are trained — 285M parameters, optimized with AdamW at a 1e-4 learning rate on 128x128x128 patches across four NVIDIA A100 GPUs.

Evaluation used a leave-one-out protocol over KiTS23 and five MSD tumor datasets, holding out one tumor type at a time. On unseen categories DiffuGTS beats SAM-based methods by at least 12.84 Dice points and the vision-language methods ZePT and Malenia by at least 4.74, reaching 59.80 Dice on held-out kidney tumors and 62.76 on hepatic vessel tumors. Trained on CT and tested on MRI — unseen tumor type and unseen modality simultaneously — it scores 50.31 Dice on an in-house liver tumor set and 44.70 on BraTS23, at least 21.28 points above every competing method.

#Applications

The practical draw is coverage without curation: a radiology group can query for a lesion type that has no annotated training set simply by naming it in a prompt, which matters most for rare tumors and for institutions whose MRI protocols differ from public benchmarks. The pseudo-healthy reconstructions are themselves interpretable, showing a clinician what the model believes the organ should look like, and the residual formulation makes the segmentation decision auditable rather than opaque.

#Impact

DiffuGTS establishes that the internal representations of medical generative models are directly usable for discriminative tasks, an argument that extends beyond tumors to anomaly detection generally. The code is released under an MIT license, though the repository has remained a stub without runnable training or inference code. The method's main cost is computational: routing every volume through MAISI for both feature extraction and generation requires 12,876 GFLOPs against 3,887 for ZePT and about 7,000 for the anomaly-detection baselines, which the authors identify as the principal barrier to deployment and a target for knowledge distillation.