MedSAM2

MedSAM2 is a promptable segmentation foundation model for three-dimensional medical images and medical videos, built by fine-tuning Meta's SAM 2.1 on a curated corpus of over 455,000 3D image-mask pairs and 76,000 annotated video frames. It was developed by Bowang Lab with collaborators at University Health Network's AI Collaborative Centre and Peter Munk Cardiac Centre, the Vector Institute, the University of Toronto, and Harvard Medical School, and posted to arXiv in April 2025.

A note on naming: two unrelated models share the name MedSAM2. This entry covers the model from Bo Wang's group (arXiv:2504.03600, repository bowang-lab/MedSAM2), the direct successor to the same lab's MedSAM. It is distinct from Medical SAM 2 (arXiv:2408.00874, repository SuperMedIntel/Medical-SAM2), a separate SAM 2 adaptation from the University of Oxford and the National University of Singapore with no shared authors.

The problem MedSAM2 targets is that most medical segmentation foundation models — including MedSAM and SAM-Med2D — operate slice by slice on 2D images, so they neither exploit volumetric continuity nor handle video. MedSAM2 unifies both under one promptable model and pairs it with the largest annotation user study reported for a model of this kind.

#Key Features

• Unified 3D and video segmentation: A single checkpoint segments CT, PET, and MRI volumes as well as ultrasound and endoscopy video, propagating a mask bidirectionally from one prompted slice or frame through the whole sequence.
• Bounding-box prompting with memory propagation: Users draw one box on the middle slice; a streaming memory bank conditions each subsequent prediction on earlier frames, keeping boundaries coherent through the volume.
• Human-in-the-loop annotation pipeline: An iterative prompt-correct-retrain loop cut per-lesion CT annotation time from 526 seconds to 74 seconds over three rounds, and echocardiogram frame annotation from 102 to 8.4 seconds.
• Lightweight backbone: Fine-tuning the SAM 2.1-Tiny variant rather than a larger one keeps compute costs modest without sacrificing accuracy, since SAM 2.1 model sizes performed statistically indistinguishably on the medical test set.
• Multi-platform deployment: Ships as a 3D Slicer plugin, a command-line tool, JupyterLab and Colab notebooks, and a Gradio web app, with Apache-2.0 code and checkpoints on Hugging Face released for research and educational use.

#Technical Details

MedSAM2 keeps the SAM 2.1 architecture — a hierarchical vision transformer (Hiera) image encoder, a prompt encoder, a memory attention module, and a mask decoder — and fully fine-tunes it, applying a lower learning rate to the image encoder to preserve pretrained features. Training data comprised 363,161 CT, 14,818 PET, and 77,154 MRI 3D image-mask pairs plus 19,232 ultrasound and 56,462 endoscopy frames, all drawn from public datasets. On a 40-task holdout set it reached median Dice scores of 88.84% (CT organs), 86.68% (CT lesions), 87.06% (MRI organs), 88.37% (MRI lesions), and 87.22% (PET lesions), beating every SAM 2.1 size and the CVPR 2024 Efficient MedSAM competition winner. On video it scored 96.13% Dice for left ventricle and 93.10% for left ventricle epicardium on CAMUS, and 92.22% on the hard polyp subset of SUN, where SAM 2.1 dropped 6-10 points. The authors note two limitations: reliance on box prompts makes thin branching structures such as vessels difficult, and a fixed eight-frame memory bank can lose targets that move rapidly or disappear and reappear.

#Applications

MedSAM2 is aimed at radiologists, cardiologists, and imaging researchers who need volumetric or temporal delineation without training a task-specific network. The released annotation pipeline produced public segmentation datasets for DeepLesion CT lesions, LUNA25 lung nodules, LLD-MMRI liver lesions, and RVENet echocardiography, each distributed on Hugging Face — a template other groups can reuse to label their own archives. The 3D Slicer plugin puts the model directly into an existing clinical research workflow, while the Colab demos let teams without local GPUs evaluate it first.

#Impact

MedSAM2's contribution is less a new architecture than a demonstration that domain fine-tuning plus disciplined data curation turns a general video segmentation model into a practical medical annotation engine, validated at a scale — 5,000 CT lesions, 3,984 liver MRI lesions, and 251,550 echocardiogram frames — that most SAM derivatives do not attempt. The segmentation masks it produced were released as public datasets, so the model's output is a reusable artifact independent of the model itself. The work remains an arXiv preprint, its weights carry a research-and-education-only restriction that limits commercial use, and inference still requires a GPU, keeping it out of point-of-care and edge deployments for now.