CT-CLIP

Every chest CT in a hospital archive already carries a radiologist's free-text report naming what is in it, written as part of routine care. A supervised 3D classifier throws that text away and trains against a label list someone paid experts to draw up, which fixes the model to a closed vocabulary and makes each new finding a new annotation project. CT-CLIP takes the opposite route: it embeds the whole volume and the whole report into a single 512-dimensional space and trains them to match, so the supervision is the sentence a radiologist had already written.

The model was built by Ibrahim Ethem Hamamci, Sezgin Er and colleagues in Bjoern Menze's group at the University of Zurich, with collaborators at ETH Zurich, Istanbul Medipol University, University Hospital Zurich, Imperial College London, Boston University, FAU Erlangen-Nürnberg and the NIH, and published in Nature Biomedical Engineering. It is one of three artifacts released together: CT-RATE, the paired volume-report corpus it learns from; CT-CLIP itself; and CT-CHAT, the conversational assistant built on its vision encoder. The contrastive recipe follows CheXzero on 2D chest radiographs, lifted to volumes.

At inference the model is given a positive and a negative sentence for each abnormality — "Consolidation is present." against "Consolidation is not present." — and a softmax over the two similarity scores yields a probability. Nothing is refit for a new dataset, and because the classifier is a sentence rather than a head, findings outside the original label set can be queried by writing them down.

#Key Features

• Zero-shot multi-abnormality detection: One fixed checkpoint scores 18 chest abnormalities on a new volume without any task-specific training, taking about 1.5 seconds per scan.
• Whole-volume encoding: Scans are standardised to 480 × 480 × 240 voxels and split into 20 × 20 × 10 patches, so cross-slice anatomy survives instead of being reduced to a slice or a projection.
• Supervision from existing records: Training consumes the findings and impression sections of routine reports, removing the manual annotation bottleneck that gates supervised 3D imaging models.
• Bidirectional case retrieval: Cosine similarity in the shared latent space retrieves comparable prior volumes from a query volume, or matching volumes from a free-text report query.
• Two fine-tuned variants: ClassFine adds a classification head for maximum accuracy on the fixed label set, while VocabFine tunes both encoders against the ground-truth logits and keeps open-vocabulary inference intact.

#Technical Details

The vision side is the encoder of CT-ViT, which tokenises a volume with 20 × 20 × 10 patches and passes them through a spatial transformer and then a causal transformer; the resulting tokens are averaged across the axial plane, flattened and projected to 512 dimensions, a compression that keeps memory low enough to train a large 3D encoder. The text side is CXR-BERT, which reads up to 512 tokens at 768 dimensions each, summed and projected into the same 512-dimensional space. A contrastive loss aligns the two projections against the identity matrix, at batch size 8 on an 80 GB A100. CT-RATE supplies 25,692 non-contrast chest CT volumes from 21,304 patients — expanded to 50,188 reconstructions, over 14.3 million 2D slices — preprocessed to 0.75 mm in-plane and 1.5 mm slice spacing with Hounsfield units clipped to ±1000. Seven prompt phrasings were compared, and "{Abnormality} is {∅/not} present." was carried forward.

Against a fully supervised CT-Net baseline retrained on the same labels, zero-shot CT-CLIP gains 0.102 mean AUROC and 0.050 mean F1 on the internal validation set, where all 18 of 18 abnormalities score higher. The margin holds under distribution shift: +0.085 AUROC and +0.073 F1 on RAD-ChestCT, and +0.087 AUROC and +0.037 F1 on a UPMC cohort, neither of which was seen during development. Training on 9.8%, 20%, 40%, 60%, 80% and 100% of CT-RATE shows accuracy still rising with corpus size.

#Applications

The zero-shot mode suits groups holding chest CT archives but no abnormality labels: cohort screening, triage prototyping, and querying for findings the label set never covered. Retrieval answers the everyday clinical question of which past cases resemble this one, from either a volume or a written description. As a frozen encoder it also serves as a 3D backbone for downstream systems, which is how CT-CHAT uses it.

#Impact

CT-CLIP is a standard baseline for open 3D chest CT encoders, benchmarked against in Merlin, CoLiPRI and RenalCLIP; X2CT-CLIP freezes it as a teacher to give a chest radiograph encoder CT-level semantics. CT-RATE anchors the VLM3D challenge series at MICCAI. The constraints are real: the corpus is single-centre and covers only non-contrast chest CT, the authors state that detection accuracy still falls short of clinical deployment thresholds, and both code and weights carry a CC BY-NC-SA licence, with the checkpoints distributed behind a click-through agreement on the CT-RATE repository.