TransDoDNet

A controller that generates a segmentation kernel for one organ at a time sees only that organ's task code and the pooled image features. Asked for the pancreas, it gets nothing about the liver, spleen, stomach and kidneys the pancreas is wedged between — yet where those organs sit is direct evidence about where the pancreas is not.

TransDoDNet keeps the dynamic segmentation head and rebuilds the thing that fills it. Seven learned organ embeddings enter a Transformer decoder as queries, attending both to the image features and to one another, so the embedding that becomes the pancreas kernel is shaped by what the liver and kidney queries have found. An MLP decodes each output embedding into the weights of a three-layer 1x1x1 convolutional head, convolved over a task-agnostic feature map to produce that organ's mask and its tumor mask. Switching targets costs nothing but a different set of filters, and all seven can be produced in parallel from one pass of the backbone.

The work is by Yutong Xie, Jianpeng Zhang and Yong Xia at Northwestern Polytechnical University with Chunhua Shen, now at Zhejiang University; part of it was done at the University of Adelaide, and Zhang is now at DAMO Academy. It extends the group's CVPR 2021 DoDNet on the same MOTS benchmark, and appeared in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2023.

#Key Features

• Transformer as kernel generator: Organ queries pass through self- and cross-attention before an MLP turns them into the dynamic head's filters, so each target's kernels are conditioned on the whole set of organs.
• Multi-scale deformable attention: Queries attend to a few learned sampling offsets across three CNN feature levels rather than to every voxel, which is what makes attention affordable on 64x192x192 sub-volumes.
• Hybrid CNN-Transformer backbone: A 3D residual encoder-decoder supplies the pre-segmentation feature map while the Transformer sits at the bottleneck, keeping the convolutional inductive bias dense prediction depends on.
• Loss over annotated targets only: Predictions for organs a scan does not label are dropped from the objective, so seven single-task corpora train one network without teaching unlabeled anatomy as background.
• Reusable 3D initialization: The pretrained CNN encoder, Transformer encoder and CNN decoder transfer to a new task by attaching a fresh head, including one in a different imaging modality.

#Technical Details

The CNN encoder-decoder is U-shaped with 50 learnable layers of 3D residual blocks and instance normalization, returning an eight-channel pre-segmentation map. At the bottleneck a three-layer Transformer encoder applies multi-scale deformable attention with eight heads over three feature levels at 192 channels, feeding a three-layer decoder carrying seven organ queries, with deep-norm residual scaling to keep the stack trainable. The generated head is three stacked 1x1x1 convolutions, eight channels wide but for a final two-channel layer splitting organ from tumor, 162 generated parameters; the whole network in this configuration is 39.05M parameters. Training used MOTS — 1,155 abdominal CT scans drawn from LiTS, KiTS and the Medical Segmentation Decathlon, 920 for training and 235 for test, re-sliced to 1.5 x 0.8 x 0.8 mm — with AdamW over 1,000 epochs and a combined Dice and cross-entropy objective. On the MOTS test set it reaches 78.26% mean Dice and 11.43 mean Hausdorff distance across 11 organ and tumor categories, against 76.94% and 12.97 for DoDNet. Ablations isolate the generator: swapping the Transformer for a convolutional controller of comparable size drops validation mean Dice from 72.30% to 70.98%.

#Applications

The immediate use is abdominal CT reading: contouring, volumetry and surgical-planning pipelines that would otherwise maintain seven organ-specific networks keep one checkpoint and choose the target at call time. The broader draw is as an initialization for cohorts with few annotations. Fine-tuned on the BCV multi-organ benchmark, the MOTS-pretrained weights reach 83.19% mean Dice on a held-out validation split, against 81.53% from scratch, 82.31% for BYOL self-supervised pretraining on unlabeled CT, and 82.51% for DoDNet's MOTS weights. Transfer also crosses modality: initializing a BraTS brain-tumor MRI network from the CT-trained weights, with the first layer's filters copied across the four MRI channels, raises average Dice from 85.74% to 86.18%.

#Impact

The claim that travels furthest is the pretraining result: strong supervision over 1,155 partially labeled scans yields a better 3D segmentation initialization than self-supervised pretraining on twice as much unlabeled CT. The limits are equally concrete. The seven organ queries are a closed set fixed at training time, so an eighth target requires retraining rather than a new query; downstream transfer is fine-tuning with a fresh head, not zero-shot generalization; and the MOTS corpus was never distributed, leaving users to rebuild it from seven upstream challenges with separate access terms. The code is GPL-3.0 and lives in a TransDoD/ subdirectory of the DoDNet repository, which ships no TransDoDNet checkpoint.