RadiSim-CL

No radiologist starts with meningioma grading. Competence arrives in order: tell an MR study from a CT, name the organ in view, point at it, call the abnormality, and only then separate a grade II meningioma from a grade I. Contrastive medical vision-language models are usually trained in the opposite spirit — every image-text pair dropped into one undifferentiated pool and sampled at random, so "an axial CT of the abdomen" and "poorly differentiated invasive adenocarcinoma" arrive together, with equal weight, from the first optimization step.

RadiSim-CL puts the order back. Developed at ShanghaiTech University's School of Biomedical Engineering with United Imaging Intelligence and published in npj Digital Medicine in June 2026, it is pretrained in three sequential phases — foundational knowledge understanding, anatomical knowledge acquisition, and advanced diagnostic reasoning — over cumulatively expanding subsets of RadiSim, a purpose-built corpus of 12 million radiology image-text pairs whose three tiers are matched to those phases. Each stage fine-tunes the previous checkpoint rather than restarting, so earlier visual-semantic structure survives into the harder material.

The deployed model is a fixed encoder pair applied zero-shot: prompts naming the candidate classes are embedded and ranked against the image embedding by cosine similarity, with no gradient updates on the user's data. That places it beside contrastive radiology models such as BiomedCLIP, PMC-CLIP, PubMedCLIP and MedSigLIP rather than generative report-writing systems like RadFM.

#Key Features

• Three-phase curriculum: Training moves from general radiological images and text, to anatomically labelled examples, to images paired with full diagnostic reports. An ablation against an identically-sized uniform-training baseline isolates the ordering as the cause of the gain on the hardest tasks.
• The RadiSim corpus: 61.18 million candidate pairs — 9.62 million from collaborating hospitals, 51.56 million from public repositories — filtered to 12 million by radiology keyword selection, LLM decomposition of compound captions, sub-image detection on composite figures, and a ResNet-101 medical/non-medical classifier.
• Sigmoid pairwise loss: The SigLIP objective replaces batch-wide softmax normalization, which penalizes semantically plausible but unmatched pairs — a common situation in radiology, where many studies share a near-identical description.
• Zero-shot organ localization: Attention maps from the image encoder's final layer are scored against organ-specific text prompts and thresholded into bounding boxes, covering 13 organ classes without detection supervision.
• Five-stage evaluation framework: 24 zero-shot subtasks as a coarse-to-fine ladder — modality recognition, anatomical recognition, anatomical localization, abnormality and disease diagnosis, and disease differentiation and grading.

#Technical Details

The architecture is an open_clip dual encoder: a ViT-B/16 SigLIP image tower at 224 × 224 resolution and a text tower initialized from BioMedBERT with a 256-token context, both projected to a 768-dimensional shared space. Pretraining used a batch size of 32,768 and a learning rate of 5 × 10⁻⁴ with 2,000 warm-up steps and cosine decay, 6–10 epochs per stage, on eight NVIDIA L40 GPUs. Evaluation sets were held disjoint from the pretraining corpus at patient level. The model reaches 0.996 accuracy on three-class modality recognition and a macro AUC of 0.888 on ten-organ MR recognition. On the reasoning tier it reports an AUC of 0.953 for brain tumour detection, 0.947 for COVID-19 on CT, 0.764 accuracy for meningioma WHO grading against 0.349 for BiomedCLIP, and 0.835 for brain tumour subtyping on an external MR dataset. The authors are explicit that the foundational and anatomical tiers are comparable to, not better than, the strongest baselines — MedSigLIP leads on brain tumour detection and on radiography pneumonia — and that the advantage is confined to fine-grained reasoning.

#Applications

The intended use is decision support and screening: sorting studies by modality and anatomy, flagging abnormal examinations, and offering a first-pass subtype or grade on brain MR and lung CT. Because classes are specified as text prompts, a site can retarget the model to a new label set by rewriting prompts rather than collecting and annotating a training cohort — the practical appeal for departments without ML engineering capacity. The authors state plainly that the model is a screening and decision-support tool and does not replace radiologist judgment.

#Impact

RadiSim-CL's contribution is evidence that ordering pretraining data by clinical difficulty, not merely scaling it, changes what a medical vision-language model can do — meningioma grading accuracy rises from 0.494 to 0.764 across the curriculum's final stage, while foundational accuracy barely moves. Its limits are equally clear: training uses static image-text pairs with no longitudinal or EMR context, the lung adenocarcinoma subtyping evaluation rests on 108 confirmed cases and is presented as preliminary, and the authors flag the risk that a many-source corpus teaches dataset artefacts alongside diagnostic features. A training launcher and a vendored copy of open_clip are on GitHub, but no code implements the three-stage curriculum; part of the RadiSim corpus is released on HuggingFace, and no checkpoint has been released — the HuggingFace model repository is gated for non-commercial academic use but currently holds no weight files.