SegAnyPET

SegAnyPET is a promptable 3D segmentation foundation model for whole-body positron emission tomography, developed at Fudan University and the Shanghai Academy of Artificial Intelligence for Science and presented at ICCV 2025. Given a PET volume and one or a few clicked points, it returns a 3D mask for the indicated organ or lesion.

PET is a poor fit for the segmentation models that work on CT and MRI. It images tracer metabolism rather than anatomy, so contrast is low, organ boundaries are diffuse, and much of the volume carries no structural detail at all. Medical adaptations of the Segment Anything family are trained overwhelmingly on structural modalities and transfer badly to this setting. The second obstacle is annotation: PET segmentation labels are scarce, and what exists is frequently incomplete or mislabeled, which makes conventional supervised training on a large corpus impossible.

SegAnyPET addresses both. The authors assembled PETS-5k, a PET segmentation corpus of 5,731 whole-body volumes covering more than 1.3 million 2D slices, and trained on it with a learning scheme that deliberately mixes a small expert-verified subset with the much larger noisy remainder rather than discarding the latter.

#Key Features

• Point-promptable across targets: A single model segments any indicated structure from positional prompts, replacing the usual practice of training one network per organ.
• Generalizes to unannotated organs: Evaluated on seven organs never labeled during training, it produces usable masks from prompts alone, with no additional annotation or fine-tuning.
• Cross prompting confident learning: Two differently prompted predictions of the same volume regularize each other, and an uncertainty-guided self-rectification step corrects labels the model is confident are wrong, so low-quality annotations contribute instead of poisoning training.
• Fully volumetric: The encoder, prompt encoder, and mask decoder are all 3D, so one prompt covers a whole structure rather than requiring a click on every slice.

#Technical Details

The architecture follows the SAM decomposition — image encoder, prompt encoder, mask decoder — rebuilt in three dimensions. The encoder splits the volume into 16 × 16 × 16 patches embedded to 768 dimensions with learnable 3D absolute positional encoding, followed by 16 3D self-attention blocks. Sparse prompts are represented by 3D positional embeddings and dense prompts by 3D convolutions with layer normalization and GELU; the decoder uses 3D transformer blocks and transposed convolutions to upscale to voxel resolution. Of the 5,731 PETS-5k volumes, 100 carry expert-verified annotations of five organs (40 for high-quality training, 60 as the internal test set) and the remaining 5,631 form the noisy training set; seven further organs were annotated on the test set to measure generalization. Training ran 200 epochs on four A100 GPUs with AdamW at a learning rate of 0.0008. Against 2D baselines including SAM and MedSAM and 3D baselines from the SAM-Med3D family, SegAnyPET improves average Dice by up to 18.40% over the second-ranked method on the internal test set, and outperforms fully supervised task-specific models while retaining promptable coverage of new targets. External validation uses AutoPET-Organ, an extension of the AutoPET dataset with 1,014 volumes and a 100-case subset annotated for all 12 organs.

#Applications

Whole-body PET reporting requires organ-level quantification — measuring tracer uptake per organ, tracking lesion burden across follow-up scans, and computing dosimetry — and each of those steps currently depends on either manual contouring or a narrowly trained organ-specific network. A promptable model collapses that into one interaction per structure, and its coverage of unannotated organs means a clinic can quantify a target the training set never included. The same interface accelerates dataset construction, since a click-and-correct loop is far faster than contouring from scratch. Follow-on releases extend the model to 11,041 multi-center volumes and add a lesion-focused variant.

#Impact

SegAnyPET establishes promptable segmentation for a modality that structural-image foundation models handle poorly, and PETS-5k gives the field a training corpus at a scale PET segmentation previously lacked. Its more transferable contribution may be the confident learning strategy: it requires no architectural change and offers a general recipe for training promptable segmenters where most available labels are imperfect. The evaluation covers organs and lesions in FDG PET, so behavior with other tracers is a separate question, and prompted segmentation still requires a human in the loop for every structure. The released checkpoints carry no stated license.