CoLiPRI

CoLiPRI (Comprehensive Language-Image Pre-training) is a family of 3D vision-language encoders that learn joint representations of chest CT volumes and their radiology reports. It targets a core bottleneck in 3D medical imaging: unlike 2D natural images, paired 3D image-text data is scarce, which limits the contrastive language-image pretraining that has driven progress in 2D vision-language models. CoLiPRI was developed by Microsoft Health Futures within Microsoft Research, with collaborators at the German Cancer Research Center (DKFZ), the University of Cambridge, Heidelberg University Hospital, and Mayo Clinic, and was introduced in an October 2025 arXiv preprint.

The model's central idea is to inject additional supervision beyond image-text alignment and to combine paired and unpaired data. Alongside a contrastive language-image objective, CoLiPRI trains a radiology report generation objective that extracts more signal from each report, and a masked image modeling objective that lets the encoder learn from image-only CT volumes for which no report exists. This combination lets a single encoder exploit both paired image-text datasets and much larger image-only collections, rather than being restricted to the limited set of scans with matched reports.

CoLiPRI is released as an open, research-only model on HuggingFace under the MIT License, with the best-performing variant (CoLiPRI-CRM) distributed as downloadable weights. It is intended for methodological research on 3D medical imaging, not for clinical use.

#Key Features

• Combined pretraining objectives: CoLiPRI jointly trains contrastive language-image alignment (CLIP), radiology report generation (RRG), and masked image modeling (MAE), extracting more supervision per scan than contrastive pretraining alone.
• Uses paired and unpaired data: The vision-only masked-image objective lets the encoder learn from image-only CT volumes, so training is not capped by the small pool of scans that have matched radiology reports.
• 3D vision transformer backbone: Built on a Primus-M vision transformer adapted for volumetric CT, producing dense representations suited to both global classification and voxel-level segmentation.
• Four encoder variants: The family spans CoLiPRI-C (contrastive only), -CR (+ report generation), -CM (+ masked image modeling), and -CRM (all three), with CRM the strongest and the released checkpoint.
• Open MIT release: Pretrained weights, source code, a demo notebook, and a model card are published on HuggingFace under the MIT License.

#Technical Details

CoLiPRI's vision encoder is a Primus-M 3D vision transformer. Pretraining used two chest CT sources: CT-RATE, contributing roughly 25,700 non-contrast CT acquisitions with paired radiology reports from Istanbul Medipol University Hospital, and the National Lung Screening Trial (NLST), contributing chest CT from approximately 26,000 patients acquired across multiple US centers and used as image-only data. The three objectives are combined in an interleaved schedule, and the released model was trained in August 2024 over about 1.75 days on four A100 GPUs. Across evaluations, a CoLiPRI encoder reaches 84.2% AUROC on CT-RATE classification probing versus 82.6% for Merlin and 61.2% for CT-CLIP, and lifts radiology report generation to a RadBERT Macro-F1 of 38.1 from 21.2 for the strongest prior 3D encoder. Zero-shot classification, image-to-report retrieval, and semantic segmentation across LiTS, Lung, HVS, and KiTS23 datasets are reported as on par with or better than prior 3D vision-language encoders.

#Applications

CoLiPRI serves researchers building 3D medical imaging systems for chest CT. Its encoder supports linear-probe and zero-shot classification of abnormalities, image-to-report and report-to-image retrieval, radiology report generation, and transfer to volumetric semantic segmentation. Because it releases open weights and inference code, it can act as a pretrained backbone for downstream fine-tuning in low-annotation settings. The authors position it strictly as a research tool intended for AI-assisted setups with human oversight, and it is not validated for clinical diagnosis.

#Impact

CoLiPRI demonstrates that adding a report-generation objective and mixing image-only with paired data can push 3D medical vision-language encoders past contrastive-only baselines such as CT-CLIP and Merlin, with a particularly large gain on report generation. By releasing MIT-licensed weights, code, and a model card, it provides a reusable starting point for the chest CT research community. Its scope is deliberately narrow: training centers on non-contrast chest CT with predominantly English reports, performance is expected to degrade on out-of-distribution anatomy or modalities, and the release is explicitly research-only rather than a clinical tool.