Pathology vision-language model that adds lightweight adaptors and multi-granular prompt learning for few-shot whole-slide image classification.
No providers recorded yet. Browse all providers
Computational pathology has two resources that do not fit together. The strongest vision encoders — models such as Prov-GigaPath, pretrained on 1.3 billion tissue tiles — are vision-only and cannot be steered with text. The pathology vision-language models that can be steered with text, such as PLIP and CONCH, were trained on far smaller image-caption corpora and inherit a general-purpose CLIP backbone. MGPATH, developed by a German-led collaboration centered on Göttingen with partners at DFKI, the University of Stuttgart, and the University of Texas at Austin, bridges the two.
Rather than retraining anything at scale, MGPATH inserts lightweight adaptor modules between a frozen large-scale pathology vision encoder and a frozen pathology text encoder, and trains only those adaptors with a contrastive objective on additional image-text pairs. The result is a vision-language model that inherits slide-scale visual representations while gaining a text interface, at a fraction of the cost of training a pathology VLM from scratch.
The downstream target is few-shot whole-slide classification, where a hospital may have a handful of annotated cases for a rare subtype and gigapixel images that cannot be labeled exhaustively. MGPATH adapts to those settings by learning prompt embeddings rather than updating the backbones.
MGPATH links the Prov-GigaPath visual encoder, pretrained on 1.3 billion pathology patches, to the PLIP text encoder, trained on roughly 200K image-text pairs, using adaptor modules optimized with a cross-alignment contrastive objective. The alignment corpus is an additional 923K pathology image-text pairs assembled from ARCH, PatchGastricADC22, and Quilt-1M. Class descriptions are generated with a frozen large language model and augmented with several learnable prompt tokens per class; these prompts are then fused with the frozen visual features at two granularities — patch level and group-of-patches level — through the multi-granular attention module, with an unbalanced optimal transport distance scoring the visual-text match. Evaluation uses three TCGA cohorts. In the 16-shot setting the PLIP-GigaPath configuration reaches AUC 93.02 ± 2.99 with 84.77 ± 4.67 accuracy on TCGA-NSCLC, AUC 98.2 ± 0.31 with 91.72 ± 1.74 accuracy on TCGA-RCC, and AUC 87.36 ± 1.85 with 79.56 ± 4.77 accuracy on TCGA-BRCA. Averaged zero-shot performance across the three cohorts is 74.67 balanced accuracy and 74.63 weighted F1. Implementations and pretrained models are released on GitHub; the repository carries no license file.
MGPATH is aimed at diagnostic settings where labeled whole-slide data is scarce: rare tumor subtypes, newly introduced biomarkers, and institutions whose archives are large but sparsely annotated. Given a text description of each class and a handful of example slides, the model produces a slide-level classifier without collecting a full supervised cohort — the regime where conventional multiple-instance learning pipelines struggle. Its adaptor recipe is also reusable as infrastructure: a group that has a strong vision-only pathology encoder can attach a text interface to it without touching the encoder weights.
MGPATH's contribution is showing that the scale advantage of vision-only pathology foundation models and the flexibility of vision-language models are combinable through parameter-efficient alignment, rather than requiring a new billion-image contrastive pretraining run. The multi-granular prompt attention addresses a real structural feature of whole-slide data — diagnosis depends simultaneously on cellular detail and tissue architecture — that flat prompt-fusion schemes ignore. The honest caveat is that adaptation still requires a few labeled examples per target class, so this is few-shot rather than truly zero-shot deployment, and the released code carries no explicit license, which constrains reuse in clinical or commercial pipelines.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.