Loading…
Medical vision-language generalist that answers questions, classifies disease, localizes findings and drafts radiology reports from one checkpoint.
No providers recorded yet. Browse all providers
A radiology report is a paragraph, a nodule localization is four numbers, and "does this patient have pneumonia?" is a single word. Because those outputs look nothing alike, medical imaging has largely built a separate model for each. MedViLaM's premise is that they only look different: write the bounding box as a coordinate string and the label as a word, and one instruction-tuned language decoder produces all three from the same image tokens.
MedViLaM was developed at the Centre for Perceptual and Interactive Intelligence at the Chinese University of Hong Kong with Shanghai AI Laboratory and SenseTime Research, and posted as a preprint in September 2024. It is assembled from parts held fixed: a ViT-L/14 visual encoder and a Vicuna-7B decoder are both frozen, and training updates only the alignment module projecting visual features into the decoder's token space. The effort goes into the instruction data rather than the backbones, and the checkpoint serves every downstream task without task-specific heads.
It sits in the generalist line alongside Google's Med-PaLM M, which likewise applies one set of weights across clinical text and imaging, and beside LLaVA-Med and Med-Flamingo. The preprint's abstract calls its corpus MultiMedBench, the name of the benchmark introduced with Med-PaLM M; the Methods instead describe a collection the authors assembled from 35 public and private datasets. The same group's CXRBase is a distinct artifact — an image-only masked autoencoder for chest radiographs, with no language decoder.
The visual encoder is a ViT-L/14 taking 896×896 inputs at a 64×64 patch size, with 1,024 hidden dimensions and 16 attention heads; a 12-layer Q-Former bridges it to the frozen Vicuna-7B decoder. Instruction tuning uses 20.5 million multi-task instruction pairs over 1.8 million medical images, covering 14 tasks across five task types and 35 datasets spanning seven modalities. Training runs 70,000 steps with AdamW under cosine annealing at a peak learning rate of 2×10⁻⁵, on 4×8 NVIDIA V100 GPUs over about two days.
On 20-shot referring-box detection across 12 datasets in six modalities, MedViLaM records the highest accuracy on ten, including 86.00 on HAM10000 dermoscopy and 30.84 on TBX11K tuberculosis, against the specialist VGTR and the generalist OFA. It scores zero on Luna16 CT lung nodules — as do both baselines, the authors attributing this to lesion size — and trails OFA on both MRI datasets. Fine-tuned on Object-CXR foreign-object detection it reaches 93.1% AUC, ahead of the 92.1% JF Healthcare baseline but below a Fast R-CNN at 95.7%. Three radiologists preferred its reports in up to 80.50% of pairwise comparisons, and judged 59% of its generated diagnoses equivalent to or better than the original physician reports.
The intended setting is a radiology service that would otherwise deploy several narrow models: draft report generation from chest films, multi-label triage, few-shot localization of findings in a new cohort, and conversational follow-up in which a clinician asks why a call was made and receives a location and severity. Adaptation to a new disease is demonstrated at 20 labelled examples, and the coordinate-string output lets a localization be checked against the image rather than accepted as a bare label.
MedViLaM shows that a generalist assembled from frozen public backbones can approach dedicated detectors on medical imaging tasks while remaining interrogable in natural language — a property that matters more in clinical review than a leaderboard position. Its practical reach is limited by what was released: the repository named in the paper's Code and Data Availability statement holds only a README and demo assets, with no training or evaluation code, and the model and dataset repositories that both the paper and that README point to on HuggingFace do not exist. The preprint has not been peer reviewed, the private evaluation cohorts are not distributable, and the authors note that the benchmark's tasks are small and its modality coverage restricted.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.