Multi-resolution vision-language foundation model for histopathology, pretrained on 34M TCGA image-text pairs across four magnifications.
No providers recorded yet. Browse all providers
A whole-slide image is a gigapixel pyramid, and a pathologist reads it by moving up and down that pyramid: nuclear morphology at 40x, glandular architecture at 10x, tissue organization at 5x. Vision-language models for computational pathology have largely ignored this, aligning image patches with text at a single magnification. MR-PLIP (multi-resolution pathology-language pretraining) is built around the alternative — pretraining across four magnifications simultaneously, with alignment losses that operate both within each resolution and across resolutions.
The model was developed at Khalifa University with collaborators at the Information Technology University of the Punjab and the University of Western Australia, and was presented at CVPR 2025. Its training corpus is roughly 34 million image-text pairs derived from about 20,000 TCGA whole-slide images — two orders of magnitude larger than the 208,414 pairs behind Stanford's PLIP, which despite the shared name stem is an unrelated model from a different group and appears in MR-PLIP's evaluation as a baseline.
The corpus is machine-annotated rather than human-annotated. Rather than harvesting paired text from the literature or social media, the authors generate captions for every patch with a frozen pathology vision-language model, which is what makes 34 million pairs tractable. The code is released under an MIT license; pretrained MR-PLIP weights have not been published.
MR-PLIP pairs a frozen UNI ViT-L/16-224 vision encoder — see UNI — with the QuiltNet text encoder, a GPT-2 tower with a 77-token context, and fuses them through a multi-modal encoder. Twenty 512x512 patches are drawn per slide at 5x, each expanded into its higher-magnification children, and captioned by Quilt-LLaVA with the prompt "Can you describe the main features visible in this histopathology image?"; the resulting descriptions are concatenated into multi-resolution text bags. Training combines standard image-text contrastive, image-text matching, and masked/prefix language modeling objectives with two additions: a cross-resolution visual-textual alignment loss over top-k keywords, and a multi-resolution text-guided visual alignment loss.
On zero-shot tile-level classification (weighted F1), MR-PLIP reaches 0.605 on BACH, 0.635 on PatchCamelyon, 0.871 on NCT-CRC, and 0.935 on DigestPath, against 0.522, 0.578, 0.803, and 0.906 for CONCH and 0.381, 0.391, 0.517, and 0.831 for PLIP. Zero-shot slide-level weighted F1 is 0.664 on CAMELYON16 and 0.875 on NSCLC-CPTAC. Under linear probing (balanced accuracy) it reaches 0.965 on NCT-CRC, 0.955 on PatchCamelyon, and 0.886 on SICAP, compared with 0.874, 0.901, and 0.826 for UNI.
The model serves computational pathology groups building slide-level classifiers, text-to-image and image-to-image retrieval over case archives, and prompt-based triage of tissue types without labeled training data. Its embeddings also work as frozen features for weakly supervised whole-slide classification, where it scores 0.950 on CAMELYON16 and 0.786 on PANDA. The repository additionally ships scripts for captioning, visual grounding, retrieval, and visual question answering built on the same aligned representation, making it a starting point for multi-modal pathology assistants rather than classification alone.
MR-PLIP demonstrates that resolution hierarchy — not just corpus size — is a usable training signal for pathology vision-language models, and that synthetic captioning can push a pathology-text corpus two orders of magnitude past hand-curated alternatives. Several caveats bound its practical reach. No pretrained checkpoint has been released, and the repository's training and inference section remains a placeholder, so the reported numbers cannot currently be reproduced end to end; the MIT license covers the code only. Captions are generated by Quilt-LLaVA and inherit whatever errors that model makes, and training exclusively on TCGA limits exposure to the staining and scanner variation of other institutions. Evaluation is retrospective and benchmark-based, with no prospective or clinical validation.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.