E3D-GPT

Whether a kidney lesion measures 1 cm or 1.5 cm is a clinical decision, and the answer lives in the physical spacing between voxels. Squash a CT volume into something a 2D encoder or a video model can read and that spacing is the first thing lost: the third axis becomes an index, not millimetres. E3D-GPT's premise is that a 3D medical vision-language model has to begin with a visual encoder built from clinical CT at a fixed physical resolution, rather than one adapted from natural video or from teaching images scraped off the web.

The model was released in October 2024 by a group at the University of Science and Technology of China, working with the Institute of Computing Technology of the Chinese Academy of Sciences, iFLYTEK's medical division, and the university's First Affiliated Hospital. Its encoder is a 3D masked autoencoder trained on 354,000 unlabeled CT volumes, all resampled to the same voxel spacing so that anatomical scale is a property the network can learn rather than an artefact of how each scan was acquired. The frozen encoder's features are then compressed by a 3D convolutional perceiver and handed to a Vicuna 7B language model adapted with LoRA.

That places E3D-GPT beside M3D, which pretrains its 3D encoder contrastively against image-text pairs, and RadFM, which handles 3D scans by treating them as video. Both are limited by the scarcity of paired 3D radiology text; the masked reconstruction objective sidesteps that constraint by requiring no captions at all, which is what lets the encoder consume a hospital archive two orders of magnitude larger than the available report corpus.

#Key Features

• Resolution-standardized 3D pretraining: Every volume is resampled to 1.5 mm in-plane and 3 mm through-plane and cropped to 224 x 224 x 112, so the encoder sees consistent physical dimensions and can reason about lesion size.
• Masked self-reconstruction at scale: A 3D adaptation of masked autoencoding hides 75% of the volume's tokens and reconstructs them, learning structural anatomy from 354K unlabeled scans without a single annotation.
• Convolutional perceiver: A single strided 3D convolution merges and projects encoder tokens for the language model, cutting 2,744 tokens to 343 while keeping neighbouring patches adjacent in the compressed representation.
• Report-derived instruction data: GPT-4 converts radiology reports into instruction-response pairs, yielding BIMCV-R-VQA (89K pairs) and CT-RATE-VQA (713K pairs) for supervised fine-tuning.
• One checkpoint, three task families: The same fine-tuned model generates reports, answers open-ended questions, and returns yes/no disease decisions from a templated prompt.

#Technical Details

The encoder is a ViT-Base backbone modified for volumetric input: patches are 16 x 16 x 8 and the embedding dimension is doubled from 768 to 1,536 to carry the larger patch content. Pretraining data comprises 300,360 volumes from a collaborating hospital plus the public BIMCV-R (6,766) and CT-RATE (47,149) collections, with Hounsfield units clipped to [-1000, 1000]. Vision-language training proceeds in two stages on 8 NVIDIA A800 GPUs: the perceiver alone is trained first, then the perceiver and the language model's LoRA adapters (r = 16, alpha = 32) together, with the visual encoder frozen throughout.

On report generation, E3D-GPT reaches BLEU-1 18.19 and ROUGE-1 23.93 on BIMCV-R against M3D's 16.43 and 21.44, and 41.15 / 52.60 on CT-RATE against 40.32 / 52.08. VQA margins are narrower — 42.24 versus 41.79 BLEU-1 on BIMCV-R — and on CT-RATE's 18-finding diagnosis task the model records 54.32 balanced accuracy and 33.76 F1, ahead of M3D's 53.37 and 33.02. An ablation swapping the convolutional perceiver for a global Q-Former, a per-patch Q-Former, MLP-Mixer, or average and max pooling puts convolution ahead of all five, with Q-Former variants the weakest.

#Applications

The intended use is drafting and querying findings from volumetric CT: producing a first-pass structured report for a chest or abdominal study, fielding follow-up questions about a specific structure or lesion, and returning binary judgements across a fixed disease list. Because the encoder is trained on brain, chest and abdominal scans from a working hospital archive rather than curated teaching cases, its features are closer to routine clinical acquisitions than those of models built from web-sourced imagery — relevant to anyone evaluating 3D medical VLMs for real radiology workflows rather than benchmark leaderboards.

#Impact

E3D-GPT's contribution is methodological: it demonstrates that a large unlabeled CT archive plus masked reconstruction produces a stronger visual substrate for a medical VLM than contrastive pretraining on the much smaller pool of paired 3D image-report data, and that a convolutional perceiver beats attention-based token merging for handing 3D features to an LLM. Its practical reach is constrained. The paper is a preprint with no journal version, and its release statement promises code and data "soon" — as of this writing no repository, checkpoint, or instruction dataset has been published, so the results cannot be reproduced and the model cannot be run. The 300K-volume pretraining corpus is private hospital data. The authors also report that hallucination persists on out-of-distribution questions, and the absolute diagnosis scores — an F1 of 33.76 across 18 findings — are far from clinical reliability.