Med-GLIP

Med-GLIP is a medical image grounding framework that links a natural language phrase to the specific region of a scan it describes — pointing at the pneumothorax when a report says "pneumothorax", not merely classifying the image as abnormal. Grounding underpins diagnostic assistance, surgical navigation, and any multimodal system that must justify an answer by showing where it looked, but the medical grounding literature has been held back by its data: existing corpora cover one or two modalities, annotate coarsely, or supply image-text pairs with no region-level supervision at all.

The work from Zhejiang University and collaborators attacks the data problem first, assembling Med-GLIP-5M, a corpus of over 5.3 million region-level annotations across seven imaging modalities, then trains a grounding framework on it. Med-GLIP adapts GLIP's reformulation of object detection as phrase grounding to medicine: a prompt such as "Detect: pneumonia, nodule, fracture" is encoded by a language model, alignment scores are computed between encoded region features and token features, and the model is trained end to end with combined classification and box-regression losses. The framework is modality-aware — each imaging modality gets a dedicated image encoder while the language encoder is shared — so that hierarchical structure, such as the distinction between a lung and a pneumonia lesion inside it, is learned from data diversity rather than from hand-designed expert modules.

#Key Features

• Seven-modality coverage: One framework spans CT, MRI, X-ray, ultrasound, endoscopy, dermoscopy, and fundus photography, where prior grounding datasets and models were typically confined to chest radiographs.
• Hierarchical region labels: Annotations run from organ boundaries down to fine-grained lesions, supporting both segmentation and grounding and allowing organ-level and multi-organ detection experiments on the same data.
• Modality-specific encoders, shared language encoder: Dedicated visual encoders absorb the very different statistics of each modality while a single text encoder keeps the phrase representation consistent across them.
• Prompt-driven detection: Because detection is posed as phrase grounding, the set of findings to localize is specified in text at inference rather than fixed by a classification head.
• Downstream spatial signal: Predicted boxes can be injected as visual features into existing report-generation and visual question answering systems without retraining those systems from scratch.

#Technical Details

Med-GLIP-5M was assembled from public repositories including GitHub, Grand Challenge, OpenNeuro, BraTS, SYNAPSE, and CodaLab, filtered for DICOM/NIfTI accessibility, spatial mask or box annotations, and anatomical coverage. The result is 2,720,979 images with an average of 4.13 masks per image, sliced from 3D volumes where necessary, standardized to PNG, and reformatted to the COCO metadata standard. A three-tier quality-control pipeline discards broken files, images whose masks fail alignment checks, malformed or semantically undefined annotations, and masks covering under 1.5% of image area. The 198 fine-grained annotation labels are reorganized into 38 anatomical categories spanning 6 major body regions. Training ran for 30 epochs with Adam at a learning rate of 2×10⁻⁴ and batch size 5 on eight NVIDIA RTX 3090 GPUs. On held-out grounding test sets, both GLIP and CO-DETR score 0.0 average precision zero-shot across all seven modalities, confirming the size of the domain gap; after full fine-tuning Med-GLIP reaches AP of 82.7 on CT (SegRap) against 31.8 for CO-DETR, 86.9 on X-ray against 69.8, 59.2 on ultrasound against 1.0, 28.0 on MRI against 7.6, and 61.7 on dermoscopy against 43.0, with CO-DETR retaining an edge on endoscopy and fundus photography. Feeding Med-GLIP's boxes into R2Gen and MLRG improves BLEU and METEOR on report generation, and into medical VQA raises closed-ended accuracy by up to 2.52% and open-ended Rouge-L by up to 1.68 on VQA-RAD.

#Applications

The immediate beneficiaries are groups building diagnostic-assistance and reporting tools who need spatial evidence attached to generated text. Because grounding outputs slot into existing pipelines as additional visual features, Med-GLIP functions as a component rather than a replacement: report generators gain region-anchored findings, VQA systems gain a way to verify that an answer corresponds to a plausible location, and interactive segmentation or triage tools gain text-promptable region proposals. The dataset itself is the broader contribution, giving the field a modality-diverse grounding corpus to train and evaluate against.

#Impact

Med-GLIP-5M is released on HuggingFace under CC BY 4.0, and the dataset card doubles as the model card, documenting architecture, intended uses, limitations, and bias risks. The trained model weights have not been released, so the grounding results are not currently reproducible from published artifacts. The authors are explicit that performance depends on the coverage of the training corpus, that generalization to rare diseases or unseen modality styles may be limited, that automated annotation noise propagates into the model, and that the system must not be used for primary clinical diagnosis without regulatory validation. The work is a preprint and has not been peer reviewed.