MerMED-FM

Clinical reasoning is rarely confined to one imaging modality. Staging a cancer may involve a CT scan and a histopathology slide; a systemic disease may present on a chest radiograph and in the retina. Yet the strongest medical imaging models are almost all single-modality specialists — RETFound for retinal images, Merlin for CT, USFM for ultrasound, UNI for pathology — each requiring its own pipeline, and each blind to the others. Attempts at unified models have tended to trade breadth for accuracy.

MerMED-FM is a self-supervised foundation model trained across seven imaging modalities at once, built to keep modality-specific fidelity while sharing a single encoder. Its distinguishing component is a dynamic memory module: a non-differentiable store of representations from earlier training iterations that current views are compared against. This gives the model a running reference for what it has already seen, which stabilizes joint training on heterogeneous data and counteracts the catastrophic forgetting that otherwise degrades earlier modalities as new ones are introduced.

The model was developed by Yang Zhou, Daniel Ting, and a large clinical and computational collaboration led from the Institute of High Performance Computing at Singapore's Agency for Science, Technology and Research, with the Singapore National Eye Centre, Duke-NUS Medical School, and SingHealth, alongside partners in China, the UK, the US, and Canada. It was posted as a preprint in June 2025 and published in The Lancet Digital Health.

#Key Features

• Seven modalities, one encoder: CT, chest radiography, ultrasound, pathology patches, colour fundus photography, OCT, and dermoscopy are embedded in a shared latent space after modality-specific preprocessing and normalization.
• Memory-augmented self-supervision: A FIFO memory of 65,536 stored representations acts as a stability anchor, preventing representation collapse and preserving knowledge of modalities seen earlier in training.
• Label-free pretraining: The entire 3.3-million-image corpus is unlabelled, sidestepping the annotation burden that constrains most medical imaging models.
• Balanced modality sampling: Each training batch is sampled to give equal weight to modalities and specialties, so that abundant data such as chest radiographs does not dominate learning at the expense of rarer domains.
• Real-world validation: Beyond curated public benchmarks, the model was tested on seven datasets from a tertiary hospital cluster in Singapore, reflecting genuine variation in equipment, demographics, and reporting.

#Technical Details

The architecture is a joint-embedding teacher-student pair with Vision Transformer backbones, following the DINO family of self-distillation methods. Twelve augmented views are generated per iteration — two global crops at 224×224 and ten local crops at 96×96 — and encoded by separate student and teacher branches, each followed by a three-layer MLP projection head with 2048 hidden units. The ViT-B encoder maps each view's [CLS] token to a 768-dimensional vector, projected to 256 dimensions on a unit hypersphere. Only the student receives gradients; the teacher is an exponential moving average of its weights. The memory holds K = 65,536 representations, partitioned into blocks of 16,384 for the similarity computation, and updates FIFO. Training used AdamW at a cosine-decayed learning rate from 1×10⁻⁵ to 1×10⁻⁶ with a batch size of 1024. The pretraining corpus comprises 1,017,712 pathology patches, 713,931 chest radiographs, 401,059 dermoscopic images, 389,885 ultrasound frames, 333,700 fundus photographs, 292,353 CT slices, and 176,719 OCT slices. For downstream use the student encoder plus an MLP head is fine-tuned per task. Reported AUROCs are 0.988 for OCT, 0.982 for pathology, 0.951 for ultrasound, 0.943 for CT, 0.931 for skin, 0.894 for fundus, and 0.858 for chest radiography, giving a mean of 0.935 against 0.933 for DINO and 0.919 for BiomedCLIP, with sensitivity of 78.3% and specificity of 91.8%.

#Applications

MerMED-FM is aimed at hospital settings where a single backbone serving many departments is operationally simpler than maintaining a separate model per specialty. Demonstrated tasks include lung cancer and COVID-19 pneumonia on chest CT, pneumonia and pneumothorax on radiographs, breast cancer on ultrasound, colorectal and breast cancer on histopathology, retinal disease on OCT and fundus photography, and skin lesion classification. Its data efficiency suits institutions with modest labelled datasets, which can adapt the pretrained encoder rather than train from scratch.

#Impact

The result of practical consequence is that a jointly trained multi-modality encoder matched or beat modality-specific foundation models on their home turf, rather than paying the accuracy penalty earlier unified attempts incurred — evidence that the breadth versus fidelity trade-off is more negotiable than assumed. Publication in The Lancet Digital Health places the work before a clinical rather than purely methodological audience. Limitations temper expectations: adaptation fine-tunes the encoder for each task rather than transferring frozen features, volumetric CT and ultrasound are handled as independent 2D slices, the ViT-B backbone is modest next to the largest pathology and radiology models, and access is constrained — the checkpoint sits behind a gated HuggingFace repository and the code carries a non-commercial CC BY-NC 4.0 licence.