CT-FM

Radiology has accumulated a large collection of task-specific models — one for cardiac segmentation, another for abdominal organs, another for triage — each with its own training run, validation burden, and maintenance cost. Foundation models offer a way out, but most radiological ones are vision-language systems whose image encoder is trained jointly with text, shaping the visual features around the vocabulary of radiology reports rather than around the fine spatial detail many tasks hinge on.

CT-FM is a vision-only alternative: a 3D encoder pretrained by contrastive self-supervision on 148,000 CT volumes from the National Cancer Institute's Imaging Data Commons, with no labels and no paired text. It was developed by the Artificial Intelligence in Medicine program at Mass General Brigham together with collaborators at Maastricht University, Brigham and Women's Hospital, and Dana-Farber Cancer Institute, and released as a preprint in January 2025.

Its neighbours make the contrast explicit. Merlin learns abdominal CT representations from reports and EHR codes; VISTA3D is supervised over a fixed class list; 3DINO-ViT mixes MRI, CT, and PET. CT-FM instead goes deep on one modality and trains purely on pixels, then shows that the resulting embeddings support four unrelated task families.

#Key Features

• Intra-sample contrastive objective: Negative views are drawn from other patches of the same scan rather than from other scans, pushing the encoder to distinguish anatomical regions within a volume instead of only telling patients apart.
• Four task families from one backbone: Whole-body and tumour segmentation, head CT triage, content-based image retrieval, and semantic concept search all run off the same pretrained weights.
• Localised, searchable embeddings: A patch query taken from one scan retrieves the corresponding anatomy in other scans, supporting concept-level search rather than whole-study similarity alone.
• Label-efficient fine-tuning: Few-shot segmentation with 5, 10, and 20 labelled examples improves significantly over training the same architecture from scratch.
• Reproducible release: Weights, training code, and the preprocessing pipeline are published through the Lighter framework, with documented steps for reproducing both pretraining and the downstream baselines.

#Technical Details

The encoder is a SegResNet, a 3D U-Net variant, with 77.8M parameters producing 512-dimensional pooled embeddings. Pretraining modifies SimCLR: several sub-volumes of 24×128×128 voxels are sampled from within a single scan, and each patch's augmented views form the positive pair while the other patches from that same scan form the negatives — an idea related to PatchNCE, used here for variance maximisation. Volumes are resampled to 2×2×5 mm spacing and centre-cropped to 128×128×48. Training ran for 500 epochs with 20 patches per scan on four NVIDIA Quadro RTX 8000 GPUs using Adam at a learning rate of 0.0001; raising the number of contrastive crops from 5 to 15 improved downstream Dice.

On TotalSegmentator's 117 anatomical structures, CT-FM reaches a mean Dice of 0.8981 against 0.8959 for the identical architecture trained from scratch, 0.8695 for the supervised SuPREM baseline, 0.882 for Auto3DSeg, and 0.893 for VISTA3D. Initialising Auto3DSeg with CT-FM weights improves Medical Segmentation Decathlon tumour Dice from 0.681 to 0.696 (hepatic) and 0.532 to 0.609 (lung), though pancreatic tumour Dice is unchanged at 0.475 versus 0.482, with the gain there appearing as a drop in average surface distance from 13.8 to 7.8 mm. Retrieval on OrganMNIST3D reaches average precision 0.932 at k=3 against 0.923 for SuPREM, and organ centroid distance for heart-patch queries falls to 5.61 cm from 23.44 cm. The head CT triage result is the exception to the paper's headline framing: on SinoCT the supervised SuPREM baseline records a higher F1 (0.798 versus 0.776) and AUC-ROC (0.868 versus 0.836) than CT-FM, which leads only in the zero-shot transfer to CQ500.

#Applications

The model is most immediately useful as an initialisation for CT segmentation and classification work, where it reduces the labelled data needed to reach a given accuracy. Its embeddings also enable content-based retrieval across a PACS archive — finding prior studies with the same organ field of view or the same lesion type — and the patch-level concept search supports auditing what a downstream model attends to. Because pretraining used the openly available Imaging Data Commons, groups can extend the cohort or reproduce the run.

#Impact

CT-FM makes a testable argument: for CT, a vision-centric encoder trained on a large unlabelled corpus produces more spatially faithful representations than encoders trained jointly with text, and the intra-sample objective is what buys the locality. The semantic-search results, where 97.7% of matched patches land inside the queried structure against 81.9% for the supervised baseline, are the sharpest evidence for that claim. The model does not dominate everywhere — a supervised segmentation-pretrained baseline still leads on in-distribution head CT triage — and the work remains a preprint. Weights, code, and cohort definitions are openly released through the Lighter framework, which lowers the barrier for other groups to reproduce the pretraining run or extend it to new cohorts.