X2CT-CLIP

Computed tomography resolves a wide range of thoracic abnormalities that a chest radiograph cannot show, but CT costs more, delivers a far higher radiation dose, and turns around slowly — which rules it out as a population screening tool. Chest radiography has the opposite profile: cheap, fast, low-dose, and available almost everywhere. Existing chest X-ray vision-language models such as CXR-CLIP and BiomedCLIP learn only from findings that are visible on the radiograph itself, so the diseases a radiologist would call from a CT have stayed out of reach.

X2CT-CLIP, developed at University Health Network with the University of Toronto and the Vector Institute, closes that gap by teaching a radiograph encoder to speak the language of a CT model. It is a tri-modal contrastive framework: the CT volume encoder and radiology-report text encoder of the pretrained CT-CLIP model are frozen, and a fresh chest radiograph encoder is trained from scratch to land in their already-aligned latent space. Because the radiograph embedding is pulled toward both the CT embedding and the report embedding of the same study, the encoder inherits CT-level disease semantics without ever seeing a CT at inference.

Prior work in this direction handled one pathology at a time — simulated-radiograph lung cancer classifiers, or the BI-Mamba cardiovascular disease model. X2CT-CLIP is the first framework to carry multi-abnormality CT knowledge across the modality gap in a single encoder.

#Key Features

• CT knowledge from a radiograph alone: At inference the model needs only a chest X-ray, yet scores abnormalities that are conventionally read off a CT volume.
• Frozen teacher, trained student: Setting the CT-report loss weight to zero keeps CT-CLIP fixed and updates only the radiograph encoder, preserving the pretrained latent structure and cutting training to a single 40GB A100 GPU.
• Cross-modal retrieval: A radiograph query retrieves its matching CT volume or radiology report, reaching R@10 of 0.181 for volume retrieval on CT-RATE against 0.055 for the CT-CLIP teacher queried with a CT.
• Backbone-agnostic: The alignment objective is independent of the radiograph architecture; both ResNet and Swin transformer encoders were trained and evaluated.
• Validated on real radiographs: Beyond simulated data, the encoder was tested on a curated MIMIC-CT set of 256 real chest X-rays paired with discharge and radiology notes.

#Technical Details

Because no public dataset pairs real radiographs with CT volumes and labels, the authors generated 50,188 (CT, report, simulated radiograph) triplets by projecting anteroposterior radiographs from CT-RATE volumes. The CT encoder is a 3D vision transformer and the report encoder a CXR-BERT, both frozen; the radiograph encoder takes 224x224 input and emits 512-dimensional embeddings. Training used an InfoNCE objective summing radiograph-report and radiograph-CT terms, 50 epochs, batch size 360, learning rate 5e-5, temperature 0.07, and AdamW. Zero-shot multi-label AUC reaches 0.716 on CT-RATE, 0.645 on RadChest-CT, and 0.567 on MIMIC-CT, above CT-CLIP itself (0.697 and 0.617) and well above radiograph-only baselines that sit near chance. Linear probing on 50% of labels gives AUC 0.847 on CT-RATE and 0.894 on RadChest-CT; external validation with a probe fit on just 5% or 10% of CT-RATE reaches 0.735 on RadChest-CT and 0.794 on MIMIC-CT. An ablation shows dropping either the report or the volume term costs about 1.5% in AUC and 2% in precision-recall AUC.

#Applications

The intended use is opportunistic screening and triage in settings where CT is scarce: a radiograph acquired for another indication can be scored for CT-diagnosable findings, flagging patients who warrant a confirmatory scan. The retrieval capability also supports report drafting and case lookup, surfacing the most similar prior CT study or report for a given radiograph. Beyond the chest, the recipe — freeze an expensive 3D foundation model, train a cheap 2D encoder into its latent space — generalizes to other 2D-to-3D alignment problems in medical imaging.

#Impact

X2CT-CLIP demonstrates that a modality's diagnostic reach is not fixed by its physics alone: supervision transferred from a richer modality can expand what a radiograph supports. The training recipe is unusually accessible, running on one GPU, which lowers the barrier for groups without large clusters. Several caveats bound the result. The work is an arXiv preprint that has not been peer reviewed, and no code, weights, or inference endpoint has been released. The preprint itself is posted under CC BY-NC-SA 4.0, so even the write-up — the only artifact on offer — bars commercial reuse and requires derivatives to carry the same terms. Pretraining relies on simulated radiographs projected from CT rather than paired real acquisitions, and the zero-shot AUCs, while ahead of every baseline tested, remain modest in absolute terms — 0.567 on the real radiograph set — so the system is positioned as a screening prior rather than a diagnostic substitute.