MMIS-Net

MMIS-Net (MultiModal Medical Image Segmentation Network) is a single segmentation model trained jointly across ten public datasets spanning 19 anatomical structures and two imaging modalities, developed in the Department of Computer Science at Brunel University London. Its headline evaluation is retinal fluid segmentation and detection on optical coherence tomography, but the point of the work is the training strategy: rather than fitting a network to one dataset, it treats the scattered archive of small, publicly annotated medical imaging datasets as one training corpus.

Combining those datasets is harder than concatenating them. The same organ carries different label indices in different collections, one dataset annotates a structure that another leaves unlabeled, and annotation protocols disagree about boundaries. Naively merged, these contradictions produce conflicting gradients and negative knowledge transfer. MMIS-Net addresses this with two mechanisms — a one-hot label space that gives every class from every dataset its own independent output, and Similarity Fusion blocks that filter feature maps down to information shared across samples.

The model is positioned against large promptable segmentation foundation models built on the Segment Anything family, including retinal and medical adaptations such as SAM-Med2D and SAMedOCT, which it outperforms on the RETOUCH benchmark while training on markedly less hardware.

#Key Features

• One checkpoint across organs and modalities: A single network handles 19 segmentation targets across CT and OCT, from liver, pancreas, spleen, and kidney tumors to the three retinal fluid classes, without per-dataset retraining.
• One-hot label space: The network shares a backbone but keeps independent per-class heads with sigmoid rather than softmax activation, so a pixel can belong to several classes at once and contradictory labels between datasets coexist instead of competing.
• Similarity Fusion blocks: Each input is smoothed at three Gaussian scales; pixels at matching positions are grouped and the one closest to the other two by Euclidean distance is kept, progressively discarding dissimilar signal to suppress negative transfer between datasets.
• Class-adaptive loss: Binary cross-entropy is combined with a Dice term computed jointly over the whole batch rather than per image, which stabilizes training when a structure is absent from some images but large in others.
• Modest compute footprint: Training took 14 hours on a single NVIDIA RTX A5000, compared with 20 hours on an A100 for the SAMedOCT baseline it outperforms.

#Technical Details

The architecture is a U-Net-style convolutional encoder-decoder with residual connections. The contracting path uses paired 3×3 convolutions with ReLU activations followed by 2×2 max pooling; the expanding path uses 2×2 transposed convolutions, skip concatenation, and further 3×3 convolutions. Similarity Fusion blocks are inserted before and after every convolutional block in both paths and in the bridge layer. Training combined ten datasets — six from the Medical Segmentation Decathlon (liver, lung, pancreas, hepatic vessels, spleen, colon) plus Pelvis, NIH Pancreas-CT, KiTS19, and RETOUCH — for 1,337 training volumes across CT and OCT. Optimization used Adam at a learning rate of 0.1 for up to 1,000 epochs with early stopping, implemented in PyTorch.

On the RETOUCH grand challenge hidden test set, held by the organizers and evaluated by private submission, MMIS-Net achieved a mean Dice score of 0.83 and mean absolute volume difference of 0.035 for segmentation of intraretinal fluid, subretinal fluid, and pigment epithelium detachment, plus an area under the curve of 1.0 for fluid detection. It obtained the best Dice in all three fluid classes and the best absolute volume difference in two of three; SAMedOCT retained the best value of 0.033 on pigment epithelium detachment.

#Applications

The immediate use is automated quantification of retinal fluid on OCT, where accurate volume estimates inform treatment decisions in age-related macular degeneration and diabetic macular edema and where manual delineation across an entire volume is impractical in clinic. More broadly, the label-space and fusion techniques offer a recipe for groups that hold several small annotated datasets covering different organs or modalities and want one model out of them rather than several, which reduces both engineering overhead and the annotation burden for any individual target.

#Impact

MMIS-Net makes a concrete case that careful handling of label inconsistency, rather than scale alone, can let a compact convolutional network beat much larger promptable segmentation models on a blind challenge test set. Its limitations should temper how it is read: training is fully supervised multi-dataset joint learning rather than self-supervised pretraining, the RETOUCH training partition was among the ten training sets so the hidden test set is a held-out partition of a seen dataset rather than an unseen benchmark, and evaluation is confined to the retinal fluid tasks even though the model was trained on 19 structures. No code or trained weights have been released, and the work is a preprint that has not been peer reviewed.