3DINO-ViT

Self-supervised pretraining transformed 2D computer vision, but 3D medical imaging has largely been left with narrow versions of the idea: pretext tasks designed for one organ, one modality, and one downstream dataset. A model pretrained on abdominal CT tends not to help with brain MRI, so each new clinical question requires its own pretraining run. The alternative — flattening volumes into stacks of 2D slices so that a general vision pipeline can be reused — discards the anatomical context that makes 3D imaging worth acquiring.

3DINO is a self-supervised framework that adapts the DINOv2 self-distillation recipe to volumetric inputs while keeping memory demands tractable, and 3DINO-ViT is the general-purpose model it produced. The team, based at the University of Toronto, Sunnybrook Research Institute, and the Vector Institute, pretrained a vision transformer on nearly 100,000 unlabelled 3D volumes drawn from 35 public and internal studies, spanning more than ten organs and three modalities. The work appeared as a preprint in January 2025 and was published in npj Digital Medicine later that year.

The distinguishing claim is generalisation rather than peak accuracy on any single benchmark. Where a radiological foundation model such as CT-FM commits to one modality at large scale, 3DINO-ViT deliberately mixes MRI, CT, and PET so that a single frozen encoder can be dropped into tasks involving organs and modalities it never saw.

#Key Features

• Genuinely volumetric pretraining: The framework operates on 3D patches with 3D positional structure rather than treating a volume as a stack of independent slices, preserving cross-slice anatomy that clinical reads depend on.
• Multimodal, multi-organ corpus: Training mixes MRI, CT, and a small brain PET cohort across more than ten organs, which is what allows one checkpoint to serve heterogeneous downstream tasks instead of a family of organ-specific ones.
• Frozen-encoder transfer: Downstream segmentation attaches a 3D ViT adapter and a convolutional decoder on top of weights that stay frozen, so the number of trainable parameters per task is small.
• Out-of-distribution robustness: The model was evaluated on an unseen organ and on an entirely unseen modality, and outperformed the compared pretrained baselines on both.
• Label efficiency: On both segmentation benchmarks, fine-tuning on under half the available labels matched baselines trained on the full labelled set.

#Technical Details

3DINO-ViT is a ViT-Large trained with a 16×16×16 voxel patch size on four NVIDIA A100 80GB GPUs, using a batch size of 512 followed by a high-resolution adaptation stage. The pretraining corpus comprises 70,434 MRI volumes, 27,815 CT volumes, and 566 brain PET volumes — nearly 100,000 scans in total, curated from 35 studies. Baselines include a randomly initialised ViT, the Swin transformer pretrained by Tang et al. and its random counterpart, and a vanilla ViT pretrained with that same framework on the identical unlabelled corpus, which isolates the contribution of the 3DINO objective from the choice of data.

Downstream evaluation covers brain tumour segmentation on BraTS 2021, 14-class abdominal organ segmentation on BTCV, left atrium MRI segmentation and 3D breast ultrasound tumour segmentation as out-of-distribution tasks, and classification on the ICBM brain-age and COVID-CT-MD lung CT datasets. With 10% of BraTS labels the model reaches 0.90 Dice against 0.87 for a randomly initialised encoder; with 25% of BTCV labels, 0.77 against 0.59. Gains narrow at full label availability, to 0.8% Dice on BraTS and 0.9% on BTCV. On frozen-feature classification it averages 18.9% higher AUC on COVID-CT-MD and 5.3% higher on ICBM than the next best baseline, and with only a two-layer linear segmentation decoder it improves on the next best baseline by 61% Dice on BTCV and 15% on BraTS.

#Applications

The model is aimed at groups building 3D imaging models where annotation is the binding constraint: rare-disease cohorts, new scanner protocols, and modalities with little public labelled data. Initialising a segmentation or classification network from 3DINO-ViT rather than from scratch reduces the annotation burden materially, and the frozen-encoder setup makes it practical to serve several downstream tasks from one set of weights. The released framework also lets other groups pretrain on their own volumetric cohorts.

#Impact

3DINO-ViT is one of the clearer demonstrations that the scaling logic behind 2D vision foundation models carries to volumetric medical imaging, and that mixing modalities during pretraining buys transfer rather than costing it. Its strongest results are in the label-scarce regime; at full label availability the advantage over well-tuned baselines narrows to under a percentage point of Dice, so the practical case is annotation efficiency rather than ceiling performance. Code and pretrained weights are released, but under a CC BY-NC-ND 4.0 license that permits neither commercial use nor redistribution of modified weights — a meaningful constraint for translational and industrial users.