MoME

Brain lesions—including tumors, strokes, white matter hyperintensities, and multiple sclerosis plaques—appear with markedly different characteristics depending on the MRI modality used to image them. As a result, automated segmentation has historically relied on task-specific models trained for a single lesion type on a single set of modalities, limiting their reuse and generalization. MoME (Mixture of Modality Experts) addresses this fragmentation by providing a single universal foundation model capable of segmenting many lesion types across heterogeneous multi-modal brain MRI.

Introduced in May 2024 by researchers at the Beijing Institute of Technology, Imperial College London, and Beijing Tiantan Hospital (Capital Medical University), MoME was early-accepted to MICCAI 2024. Its central idea borrows from the Mixture of Experts paradigm: rather than forcing one network to learn every modality at once, MoME assembles a team of expert networks, each specialized for a particular imaging modality, and combines their predictions through a learned gating mechanism. This design lets the model exploit modality-specific cues while still producing a unified segmentation.

By treating lesion segmentation as a universal rather than narrow task, MoME fits into the broader move toward generalist medical-imaging foundation models—models intended to serve as reusable backbones across many clinical datasets and acquisition protocols rather than being retrained from scratch for each new study.

#Key Features

• Mixture of Modality Experts: Multiple expert networks each attend to a particular MRI modality (T1, T1ce, T2, FLAIR, DWI), enhancing capacity by letting each expert specialize in the appearance of lesions on its modality.
• Hierarchical gating network: A learned gating module combines expert predictions and fosters collaborative expertise exploration, so the model can adaptively weight experts for the modalities present in a given scan.
• Curriculum learning strategy: Training follows a curriculum that prevents individual experts from degenerating and preserves their specialization, a key ingredient for keeping the mixture meaningful.
• Universal coverage: A single model segments eight lesion types across five imaging modalities, evaluated on nine brain lesion datasets and 17 tasks.
• Generalization to unseen data: The authors report promising performance on datasets not seen during training, an important property for clinical deployment across sites and scanners.

#Technical Details

MoME is built on the widely used nnU-Net segmentation framework, with the mixture-of-experts structure layered on top of convolutional U-Net backbones. Each modality expert is a segmentation network, and a hierarchical gating network fuses their outputs; a curriculum learning schedule maintains expert specialization during joint training. The model was developed and evaluated using nine public and in-house brain lesion datasets—including BraTS, ATLAS, OASIS, ISLES, WMH2017, and MSSEG—spanning five MRI modalities and eight lesion types across 17 segmentation tasks. The authors report that MoME outperforms state-of-the-art universal segmentation models across different modalities and lesion types, while also generalizing to unseen datasets. Pretrained checkpoints for the modality experts and the full MoME model are released, and the code is available under an Apache-2.0 license.

#Applications

MoME targets neuroimaging research and clinical workflows where brain lesions must be delineated across diverse MRI protocols, such as tumor volumetry, stroke lesion quantification, multiple sclerosis lesion load tracking, and white matter hyperintensity assessment. Because one model handles multiple modalities and lesion types, it is well suited to multi-site studies and heterogeneous clinical archives where acquisition protocols vary, reducing the need to build and maintain a separate segmentation pipeline for each lesion type or scanner. Radiology researchers and medical image analysis groups benefit from a reusable backbone that can be applied or fine-tuned across many lesion segmentation tasks.

#Impact

MoME contributes to the growing effort to build universal foundation models for medical image segmentation, demonstrating that a modality-aware mixture of experts can outperform single-network universal models on brain MRI. Its public Apache-2.0 code and released checkpoints lower the barrier for groups working on brain lesion analysis to adopt or extend the approach. The work was subsequently extended into a peer-reviewed journal version, "A Foundation Model for Lesion Segmentation on Brain MRI With Mixture of Modality Experts," published in IEEE Transactions on Medical Imaging (vol. 44, no. 6, pp. 2594–2604, 2025), which adds handling of combined multi-modality inputs via a soft-assignment dispatch network. Checkpoints for this extended model (MoME+) are distributed alongside the original on the project's GitHub and Hugging Face repositories, signaling continued development of the MoME line beyond the original MICCAI conference paper.