Loading…
Vision-language model for annotation-free pathology localization, marking the finding a text prompt names in X-ray, histology and fundus images.
No providers recorded yet. Browse all providers
A radiologist who writes "tiny right basilar pneumothorax" has already localized the finding — in words. A contrastive model that scores one whole image against one whole report throws that away: "right basilar" is averaged into a single report embedding alongside every other sentence, and the model learns that the radiograph matches the document without ever learning which pixels the phrase referred to. Supervised localization recovers the missing signal by paying experts to draw boxes, the bottleneck that confines such systems to the pathologies someone was funded to annotate.
AFLoc, from Shanshan Wang's group at the Shenzhen Institutes of Advanced Technology with collaborators across eleven other institutions, closes the granularity gap by aligning three levels of text against three depths of image feature simultaneously. Words are matched to shallow local features, sentences to deeper local features, and the report as a whole to the pooled global feature, so the fine-grained spatial language already in clinical reports supervises the fine-grained visual representation. At inference the model becomes a text-prompted detector: it scores an image against a phrase such as "findings suggesting consolidation", classifies, and emits a heatmap over the region driving that score — no annotations, no fine-tuning, no fixed label set.
Where CheXzero showed that report-driven pretraining reaches radiologist-level classification, AFLoc extends the bargain to localization.
The image encoder is a ResNet-50, with shallow local features taken from the third downsampling stage, deep local features from the fourth, and a global vector from average-pooling the final convolutional layer; a projection layer maps all three to the 768-dimensional text space. The text encoder is BioClinicalBERT, whose last four layers are averaged into subword features and aggregated into word, sentence, and report representations. Chest X-ray pretraining used 220,000 image-report pairs from MIMIC-CXR, taking text from the Findings and Impression sections, with Adam at a learning rate of 2×10⁻⁵ decayed by 0.9 per epoch. Histopathology pretraining used Quilt-1M; the retinal model used an in-house set of 26,028 fundus images from 11,372 individuals.
Across eight external chest X-ray datasets covering 34 pathologies, AFLoc reaches IoU 0.324 on MS-CXR against 0.268 for GLoRIA, 0.264 for MedKLIP, and 0.228 for BioViL, and IoU 0.342 with CNR 1.538 on RSNA Pneumonia. On CheXlocalize it averages IoU 0.318 against Grad-CAM's 0.282 and exceeds the human benchmark on five pathologies. Zero-shot classification gives AUROC 0.881 on RSNA, 0.902 on SIIM-ACR, and 0.737 on NIH ChestXray14. On histopathology it reaches IoU 0.285 on SICAPv2 against 0.249 for PLIP, the strongest of the compared histopathology models, which also included CONCH and BiomedCLIP. Across nine retinal diseases its mean AUROC of 0.908 equals RETFound, which was fine-tuned on human labels.
The practical draw is querying findings nobody annotated: a hospital can prompt for a rare pathology or an emerging disease without assembling a labeled cohort first. In a controlled reader study, two board-certified radiologists scoring 100 MIMIC-CXR images across 14 binary decisions improved from 10.80 to 11.74 with AFLoc's heatmaps alongside the image, while average reading time fell 20.5% from 27.92 to 22.20 seconds. Where labels do exist, fine-tuning on 1% of the SIIM training set reaches a Dice of 0.772.
AFLoc demonstrates that report text carries enough spatial structure to supervise localization, provided the alignment is built at matching granularity — a design lesson that transfers to any domain where free-text descriptions accompany images. The paper states its limitations plainly: localization is emitted only for positive classifications, so a missed diagnosis is a missed localization; diffuse findings remain hard, with intraretinal hemorrhage reaching only IoU 0.097; and modalities beyond the three tested are unexplored. Reproducing pretraining is gated in practice, since MIMIC-CXR requires credentialed PhysioNet access and the retinal fundus corpus is not public. The code is Apache-2.0 and all three pretrained checkpoints — chest X-ray, histopathology, and fundus — are distributed through a Google Drive folder rather than a versioned model hub.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.