KEEP

Pathology vision-language models learn by contrasting image patches against their captions, a recipe borrowed intact from CLIP and used to good effect by PLIP and CONCH. The recipe has a structural blind spot: it treats every caption as an independent string. "Invasive ductal carcinoma" and "invasive breast carcinoma of no special type" are the same diagnosis under two names, and each is a subtype of breast carcinoma, but a contrastive objective sees three unrelated captions and is trained to push them apart. Rare subtypes suffer most, because they appear too infrequently for the model to infer these relationships from co-occurrence.

KEEP — KnowledgE-Enhanced Pathology, developed by the MAGIC-AI4Med group at Shanghai Jiao Tong University and Shanghai AI Laboratory — supplies that structure explicitly. It builds a disease knowledge graph covering 11,454 human diseases and 139,143 attributes, including synonyms, definitions, and hierarchical parent-child relations, and uses it to reorganize millions of noisy pathology image-text pairs into roughly 143,000 semantic groups. Each group holds a single canonical caption paired with many images that the ontology says depict the same disease entity. Alignment then happens against a semantic hierarchy rather than against raw caption strings.

The practical payoff is zero-shot diagnosis: a pathologist can present a whole-slide image with a list of candidate diagnoses written in plain text and get a ranking back, without curating a labeled training set for that cancer type. That distinguishes KEEP from vision-only pathology encoders such as UNI, which require a trained classifier head for each new task.

#Key Features

• Knowledge-graph restructured pretraining: Image-text pairs are grouped by disease identity before contrastive learning, so synonymous captions reinforce rather than compete.
• Hierarchical semantic alignment: Visual and textual representations are aligned within an ontology hierarchy, placing subtypes near their parent disease in the embedding space.
• Zero-shot across the diagnostic pipeline: The same checkpoint performs cancer region segmentation, whole-slide cancer detection, and subtyping from text prompts alone.
• Rare-subtype generalization: The knowledge graph carries relational information the image corpus cannot supply for diseases with only a handful of examples.
• Loads through the standard stack: The released checkpoint runs under Hugging Face Transformers with separate image and text encoding calls, and ships under an MIT license.

#Technical Details

KEEP pairs a ViT-L/16 vision encoder — 24 layers, hidden size 1,024, 16 heads, 224×224 inputs — with a 12-layer BERT text encoder at hidden size 768, projecting both into a shared 768-dimensional space, for roughly 414 million parameters in total. The disease knowledge graph draws 11,454 diseases and 139,143 attributes from disease ontologies, and the restructuring collapses millions of public pathology image-caption pairs into about 143,000 semantically structured groups.

Evaluation covers 18 public benchmarks with more than 14,000 whole-slide images, plus 4 institutional rare-cancer datasets totaling 926 cases. On zero-shot cancer detection, KEEP reaches an average sensitivity of 89.8% at 95.0% specificity across 7 cancer types, with AUROC of 0.999 on CPTAC-CCRCC against 0.871 for CONCH, and 0.929 on CPTAC-PDA against 0.920. Zero-shot cancer region segmentation gives Dice scores of 0.361 on CAMELYON16 and 0.530 on AGGC22, versus 0.292 and 0.449 for CONCH. On zero-shot subtyping, balanced accuracy reaches 0.774 on TCGA-BRCA and 0.604 on TCGA-BRAIN, against 0.727 and 0.453; on EBRAINS, covering 30 rare brain cancer subtypes, it reaches a median balanced accuracy of 0.456 against 0.371.

#Applications

The clearest fit is diagnostic triage on whole-slide images where labeled data is the bottleneck: rare tumor subtypes, second-opinion workflows, and retrospective cohort screening. Because prompts are ordinary text, a pathology group can adapt the diagnostic vocabulary to a local classification scheme without retraining anything. The segmentation results also make it usable as a region-of-interest finder ahead of manual review, and the released embeddings serve as features for downstream slide-level models.

#Impact

KEEP argues that curated medical knowledge is a training signal, not just an evaluation resource, and that ontological structure is what lets a pathology foundation model handle the long tail of rare diagnoses. The work has been published in Cancer Cell, and the MIT-licensed weights and code have seen substantial uptake. The supporting data has been released since publication: the disease knowledge graph, the 1,000 manually annotated pathology images and the pathology image-text semantic groups used for training are distributed as a Hugging Face dataset, and the institutional pediatric rare-cancer slides are distributed as KidRare, 2,331 whole-slide images spanning neuroblastoma, nephroblastoma, medulloblastoma and hepatoblastoma, with the KEEP subtype labels alongside. Both dataset repositories are click-through gated and restricted to non-commercial academic use, KidRare additionally under CC BY-NC-ND 3.0, and neither carries a written data card — so the training distribution is reconstructable from the released groupings but not documented as such.