Loading…
Histopathology vision-language model classifying H&E patches zero-shot from text prompts, trained on 1.6M captions written for whole-slide crops.
No providers recorded yet. Browse all providers
A pathology CLIP model is only as good as the captions it reads, and until recently those captions came from whatever the internet happened to have paired with a tissue image: a compressed figure panel in a PubMed article, a screenshot from a conference talk uploaded to YouTube, a tweet whose text admires the staining rather than describing the morphology. The largest such corpora top out below a million pairs, and the images are degraded copies of the high-resolution scans pathologists actually look at. Meanwhile TCGA holds thousands of diagnostic-quality whole-slide images — with slide-level reports, but no text attached to any individual patch.
PathGen-CLIP is built on the observation that those captions can be manufactured rather than scavenged. A cascade of agent models works over each slide: prompt-based cross-modal retrieval and k-means clustering select 384 representative patches, a pathology-tuned describing model writes a detailed caption for each, a revision model corrects hallucinated or contradictory statements, and a summarization model compresses the result to fit CLIP's 77-token text encoder. Run over roughly 7,300 TCGA slides spanning 27 tissue types, the pipeline yields PathGen-1.6M, 1.6 million image-caption pairs, and PathGen-CLIP is the contrastive model trained on it.
The work comes from Westlake University with collaborators at Zhejiang University and The Ohio State University, the same group behind PathAsst and CPath-Omni, and was published at ICLR 2025.
Both variants are trained with the OpenCLIP framework: a ViT-B/16 base model and PathGen-CLIP-L at ViT-L/14 with 336-pixel inputs, whose released weights carry roughly 428 million parameters across the vision and text towers. Across nine external patch-classification datasets — PatchCamelyon, CRC-100K, SICAPv2, BACH, Osteo, SkinCancer, WSSSLUAD, LC-Lung and LC-Colon, none of them TCGA — zero-shot accuracy averages 74.3% for PathGen-CLIP and 79.7% for PathGen-CLIP-L, against 68.7% for CONCH, 66.2% for PathCLIP, 58.9% for QuiltNet, 58.6% for PLIP and 52.9% for BiomedCLIP. Under linear probing with only two labelled examples per class, PathGen-CLIP-L approaches 92% on WSSSLUAD.
For whole-slide classification on CAMELYON16, CAMELYON17 and BRACS, the frozen encoder feeds ABMIL and ACMIL aggregators. PathGen-CLIP-L averages 92.6 AUC under ACMIL, ahead of CONCH at 91.1 and of the GigaPath tile encoder from Prov-GigaPath at 90.9, despite that encoder having been pretrained on over a billion patches from private data. Two pathologists auditing the generated captions rated their per-caption accuracy at 90.3% and 88.5%.
The zero-shot path suits screening and triage settings where a new tissue or lesion type arrives without any labelled cohort: the task is defined by writing class prompts. The few-shot path covers the common case of a handful of annotated slides per class. As a frozen tile encoder it slots into existing multiple-instance learning pipelines for subtyping and metastasis detection, and its text tower supports cross-modal retrieval — pulling patches matching a described finding out of a slide. It also serves as the vision tower for PathGen-LLaVA, the conversational model released alongside it.
The substantive result is that captions produced by a captioning model, applied to high-quality slides, beat captions harvested from the web — a scalable route past the ceiling that pathology vision-language work had been sitting under. The training corpus is public, but as coordinates and captions only: reproducing it means downloading roughly 7,300 TCGA slides through the GDC client, and the dataset repository is gated behind non-commercial research terms. Licensing across the artifacts is inconsistent, with the base checkpoint declared CC BY 2.0, the large checkpoint CC BY-NC 4.0, its HuggingFace conversion CC BY 4.0, and the code repository carrying no license file at all. The captions are machine-written, so systematic errors in the describing model propagate into everything trained on them, and because the source is TCGA the corpus skews toward malignancy.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.