Pathology vision-language model whose text encoder is aligned to 335,645 clinical reports over a frozen CONCH v1.5 image encoder, for zero-shot use.
No providers recorded yet. Browse all providers
A pathologist signing out a case writes "metastatic breast carcinoma in one of three sentinel lymph nodes", not a caption describing each field of view. That mismatch is why pathology vision-language models have been trained on curated image-caption pairs scraped from textbooks, publications and educational video: a report describes a whole slide's diagnosis, while a patch encoder sees only a few hundred microns of tissue at a time. Curated captions are scarce, though, typically around a million pairs, while diagnostic reports are produced for every case a hospital processes. CONCH-Z is a model trained to learn from the reports anyway.
It was developed by the Mahmood Lab at the Mass General Brigham AI Institute and Harvard, with collaborators at MIT and the University of Augsburg, and first posted in October 2026 as a NeurIPS 2026 Spotlight paper. CONCH-Z is derived from CONCH v1.5: its image encoder is the CONCH v1.5 vision tower, left entirely frozen, and only the text encoder is retrained. The training recipe, Locked Image Multiple Instance PreTraining (LIMIT), treats each slide as a bag of patch embeddings and uses the embedding of a report as a cross-attention query over that bag. The resulting report-conditioned slide vector is then matched against the correct report and against hard negative reports from the same organ.
The result keeps the patch-level image space of its parent, so existing CONCH v1.5 feature caches stay valid, while the text side now speaks the language of clinical diagnosis. The same text encoder can score individual patches or, through attention pooling, whole slides.
conch_v15
features need only a 768x768 projection to be compared with CONCH-Z text embeddings.The released checkpoint holds about 418 million parameters: the CONCH v1.5 ViT-L/16 vision encoder with its attentional pooler, CONCH v1.5's frozen 768x768 contrastive projection, and the newly aligned text encoder producing 768-dimensional embeddings. Training used 335,645 H&E and immunohistochemistry whole-slide images paired with their surgical pathology reports from the archives of Massachusetts General Hospital and Brigham and Women's Hospital, spanning 20 organs; it is the same slide collection used to train TITAN. Reports were de-identified and normalized with a locally hosted Qwen3-14B to strip details not visible in histology. Training ran for three epochs at a learning rate of 1e-5 on precomputed 448x448 patch features, with an InfoNCE temperature of 0.07. Across 22 zero-shot tasks, CONCH-Z reached 96.7 AUC for CAMELYON-16 metastasis detection and 58.8% balanced accuracy on the coarse SPIDER tissue taxonomy, versus 90.3 and 49.7 for CONCH v1.5. On 45-class TCGA OncoTree subtyping it reached 78.8% balanced accuracy, against 73.1% for TITAN and 57.4% for its frozen base. Ablations show the similarity weighting is worth about 8 points and that full fine-tuning of the text encoder beats LoRA.
CONCH-Z is aimed at researchers who need to label pathology images without training a classifier: prompting patches for tissue types, screening lymph node sections for metastasis, or subtyping a slide among the diagnoses possible for its organ. Because text queries pool patches by similarity, a prompt such as "breast cancer metastasis" also yields a heatmap of the regions that drove the call, which the paper shows overlapping pathologist annotations. Groups with existing CONCH v1.5 feature caches can add zero-shot and text-retrieval capability without re-extracting a single tile.
CONCH-Z argues that pathology language supervision does not need to be curated, and that slide-level text can sharpen patch-level alignment rather than requiring a dedicated slide encoder. Its slide-level results come from a model with no slide-specific parameters, narrowing the gap to supervised multiple-instance baselines such as ABMIL and outperforming them in end-to-end zero-shot use on TCGA. The approach has stated limits: because no image-side parameters are learned, global slide properties such as grade or tumor extent are harder to capture, and evaluation covers diagnosis, subtyping and tissue classification only. Weights are gated on HuggingFace under CC-BY-NC-ND 4.0 for non-commercial academic research, LIMIT training code has not been released, the training reports are private institutional data, and the model is not validated for clinical diagnosis.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.