RenalCLIP

RenalCLIP is a three-dimensional vision-language foundation model for the interpretation of renal masses on computed tomography, built by the Digital Medical Research Center at Fudan University together with Microsoft Research Asia and urology and radiology departments at nine Chinese medical centers. It targets a specific clinical problem: renal masses are increasingly found incidentally, and distinguishing an indolent or benign lesion from an aggressive cancer without a biopsy remains unreliable. Roughly one in five surgically resected renal masses is ultimately confirmed benign, which means patients undergo operations they did not need.

The model's design is deliberately disease-centric rather than general-purpose. Instead of aligning a generic CT encoder to arbitrary radiology text, RenalCLIP uses a two-stage pretraining strategy: each encoder is first infused with kidney-cancer-specific knowledge on its own, and only then are the two aligned with a contrastive objective. The result is a shared image-text embedding space tuned to the vocabulary and anatomy of renal oncology.

RenalCLIP sits alongside general-purpose CT foundation models such as CT-CLIP, Merlin, and CT-FM, which it uses as baselines across the full kidney-cancer workflow. Code is released under an MIT license on GitHub, with pretrained image- and text-encoder weights on Hugging Face.

#Key Features

• Two-stage, knowledge-enhanced pretraining: The image and text encoders are each specialized on domain data before contrastive alignment, which the authors show is what separates RenalCLIP from encoders aligned from scratch.
• Zero-shot diagnosis: Using prompt ensembles, the model classifies malignancy and aggressiveness with no task-specific fine-tuning, and on the external cohort its zero-shot malignancy prediction exceeds the fine-tuned performance of every baseline.
• Cross-modal retrieval: A shared embedding space supports text-to-image and image-to-text search over CT volumes and their reports, useful for cohort identification and teaching.
• Structured report generation: A report head built on the frozen encoders scores highest on BLEU, METEOR, and ROUGE-L against GPT-4o, MedGemma, RadFM, and CT-CHAT.
• Data-efficient adaptation: On diagnostic classification, fine-tuning on 20% of the labeled data matches the peak performance of baselines fine-tuned on all of it.
• Released weights with a gated dependency: The image encoder ships as a full checkpoint, but the text encoder is distributed only as LoRA adapters, so users must separately download the Llama-3-8B-Instruct base model to reconstruct it.

#Technical Details

The image encoder is a 3D ResNet-18 backbone with a projection layer, pretrained through a multi-task framework that predicts renal-mass attributes parsed from radiology reports. The text encoder adapts the 32-layer Llama-3 8B model into a sentence embedder using LLM2Vec: masked next-token prediction with LoRA on 227,973 MIMIC-CXR reports, followed by SimCSE contrastive refinement on an in-house kidney cancer corpus. Training used 27,866 CT scans from 8,809 patients across nine Chinese centers plus the public TCIA cohort, with free-text reports parsed into structured attributes by a large language model.

Across ten tasks, RenalCLIP led on anatomical R.E.N.A.L. nephrometry scoring (external ROC AUC 0.902 for radius, 0.727 for location), malignancy diagnosis (external ROC AUC 0.841 versus 0.739 for CT-FM), and recurrence-free survival on TCIA (C-index 0.726 versus 0.592), where its risk score remained an independent prognostic factor after adjusting for TNM stage and WHO/ISUP grade (p=0.016, HR=2.27). Zero-shot malignancy AUC reached 0.730 against 0.664 for Merlin and 0.605 for CT-CLIP, and mean text-to-image Recall@5 was 0.120 against 0.051 and 0.004 respectively.

#Applications

The model is intended as a single backbone for the kidney cancer imaging workflow: automated nephrometry scoring to support surgical planning, non-invasive triage of benign versus malignant masses, aggressiveness stratification, draft report generation, and pre-operative risk scores that could enrich clinical trials or guide surveillance versus resection decisions. Its data efficiency matters most for centers with small annotated cohorts, which can adapt it to a local task without assembling thousands of labeled scans.

#Impact

RenalCLIP demonstrates that narrowing a vision-language foundation model to one disease, rather than to one imaging modality, can outperform much broader CT models on every stage of that disease's workflow — including prognosis, where general encoders failed to produce statistically significant risk scores. The open MIT release of code and weights makes it a usable starting point for renal imaging research. Its limitations are equally concrete: the study is retrospective, the cohorts are drawn from Chinese centers plus one public American dataset, the report-generation and survival results have not been validated prospectively, the text encoder cannot be used without access to a separately licensed Llama-3 base model, and the work remains a preprint pending peer review.