CT-CHAT

A radiologist reading a chest CT scrolls through several hundred axial slices, and the meaning of a single nodule depends on which lobe it sits in and on what is happening a dozen slices away. A vision-language assistant built for 2D radiographs cannot hold that: given one slice, or a digitally reconstructed radiograph of the study, it answers about a projection rather than about the scan the clinician read. CT-CHAT keeps the volume, passing the full 3D study through a pretrained volumetric encoder before any language model sees it.

CT-CHAT was developed by Ibrahim Ethem Hamamci, Sezgin Er and colleagues in Bjoern Menze's group at the University of Zurich, with collaborators at ETH Zurich, Istanbul Medipol University, Imperial College London, Boston University and the NIH, and published in Nature Biomedical Engineering. It is the conversational component of a three-part release that also includes CT-RATE — 25,692 non-contrast chest CT volumes from 21,304 patients, paired with their radiology reports — and CT-CLIP, the contrastive encoder trained on them.

Architecturally it is a 3D adaptation of the LLaVA recipe: the frozen CT-CLIP vision transformer compresses a volume into CT tokens, a multimodal projector maps those into a language model's embedding space, and the language model — instruction-tuned once on 2.7 million question-answer pairs derived from CT-RATE — generates the reply. Nothing is refit per study.

#Key Features

• Volumetric visual input: Each scan is resampled to 480 × 480 × 240 voxels and encoded by the frozen CT-ViT transformer inherited from CT-CLIP, so the model reasons over cross-slice anatomy rather than a projection.
• Four response modes: The tokens <long_answer>, <short_answer>, <report_generation> and <multiple_choice> are attached to the training conversations and used at inference to switch between discursive explanation, terse answers, drafted findings, and exam-style selection.
• Attention-pooling projector: 256 learned latent queries cross-attend to the encoder's final layer, compressing the volume into a fixed-length token sequence regardless of slice count.
• Structured nodule conditioning: Pulmonary-nodule masks from TotalSegmentator supply lobar location, centroid, orthogonal diameters and volume in a fixed prompt block, lifting report F1 on the internal validation set from 0.184 to 0.305.
• Hallucination guardrail: A system prompt suppresses confident pathology claims when no volume accompanies the question, an ablation-driven safeguard for the text-only case.

#Technical Details

The primary configuration pairs the encoder with 70B Llama 3.1 — 80 layers, 64 attention heads, 8,192-dimensional embeddings, 128k context — with 8B Llama 3.1, Vicuna 13B and Mistral 7B variants trained the same way; the ablation found the backbone choice mattered far less than whether the volume was supplied. Training runs in two stages: a medical-concept alignment step trains only the projector to produce radiology reports with the language model frozen, then instruction tuning updates the projector together with LoRA weights (rank 128, alpha 256) on the CT-VQA corpus — 281,689 long-answer, 318,710 short-answer, 46,911 multiple-choice and 47,149 report-generation conversations, derived from CT-RATE reports mostly with Llama 3.1 8B.

Against 2D assistants evaluated on digitally reconstructed radiographs — LLaVA 1.6 in its Vicuna 13B and Mistral 7B forms, LLaVA-Med and CXR-LLaVA — CT-CHAT scores higher on BLEU-1, METEOR, ROUGE-L, CIDEr and an LLM-judged clinical accuracy score across all four task types. Report generation was benchmarked separately against RadFM and CT2Rep using clinical-efficacy metrics: F1 0.184 and CRG 0.368 on the CT-RATE internal validation set, against 0.059 and 0.335 for RadFM and 0.160 and 0.359 for CT2Rep, with the ordering preserved on the external RAD-ChestCT set. Three board-certified radiologists scoring 150 stratified reports per model on a 0–10 scale rated CT-CHAT at 3.44, 4.10 and 4.52, against 2.66, 2.52 and 2.74 for CT2Rep.

#Applications

CT-CHAT is aimed at the reading room and the research settings around it: putting targeted questions to a scan, drafting a first-pass findings section for a radiologist to correct, and answering short-answer or multiple-choice queries for teaching and dataset curation. The repository ships a Gradio interface, so a clinical group can load a volume and converse with it without an ML engineer. Because the model accepts structured measurements in the prompt, it also works as a language front end to an existing 3D imaging pipeline.

#Impact

Releasing CT-CHAT alongside CT-RATE and CT-CLIP under a CC BY-NC-SA licence gave the field an open reference point for 3D radiology question answering, and it now serves as a baseline in later multimodal medical work such as RenalCLIP. The limits are stated plainly by the authors: training data comes from a single hospital in Istanbul, coverage is restricted to non-contrast chest CT, and the radiologists' scores — all under 5 out of 10 — put report quality well below what routine clinical use demands. The non-commercial licence, and the licence terms of the Llama, Vicuna and Mistral backbones, constrain redistribution accordingly.