PRISM2

Computational pathology's first wave of foundation models — tile encoders such as Virchow, UNI, and Prov-GigaPath — learn from image patches with self-supervised objectives, which is powerful but supplies no clinical grounding. A slide contains hundreds of thousands of tiles; deciding which patterns matter for a diagnosis is precisely the judgment that pathologists encode in their reports, and that signal goes unused when pretraining is purely visual.

PRISM2, from Paige with Microsoft Research, Memorial Sloan Kettering Cancer Center, and Yale University, supervises slide-level pretraining with the language of diagnostic reasoning instead. Its training set pairs 685,507 clinical specimens — 2.3 million whole-slide images from 200,692 patients — with the reports written about them, expanded into roughly 14 million question-answer pairs. Learning to answer those questions forces the slide representation to align with the morphologic features a pathologist actually names.

The consequence is a model that works two ways at once. It can be prompted directly, answering diagnostic questions or generating a report from a slide, and it emits transferable slide-level embeddings that feed conventional downstream classifiers. First posted as a preprint in June 2025 and published in Nature Medicine in 2026, it succeeds Paige's earlier PRISM slide encoder and is released on HuggingFace for non-commercial research.

#Key Features

• Clinical-dialogue supervision: Rather than contrastive image-text matching alone, training expands each report into structured question-answer exchanges, giving a denser and more clinically grounded signal than a single caption per slide.
• Two embedding types: A 2560-dimensional "base" embedding from the perceiver pooling layer captures general slide content, while a 3072-dimensional "diagnostic" embedding taken from the language model's hidden state is tuned toward diagnostic distinctions.
• Prompted inference without retraining: Yes/no scoring, multiple-choice answering, and full report generation run directly off a slide with no task-specific head or labeled fine-tuning set.
• Clinical-grade cancer detection off the shelf: Prompt-based inference matches or exceeds the balanced accuracy of deployed clinical products calibrated for prostate, breast, and breast lymph node cancer detection.
• Fine-tunes further where data allows: Survival-specific fine-tuning on a large outcomes dataset beats specialist models trained from scratch on the same data.

#Technical Details

PRISM2 stacks three components. A frozen Virchow2 tile encoder converts 224x224 tiles at 0.5 microns per pixel into 1280-dimensional class-token embeddings. A trained perceiver slide encoder — one cross-attention block between 256 latent queries and the tile embeddings, followed by six self-attention layers, with attention pooling on the output — aggregates them into a slide representation. That representation conditions Phi-3 Mini, a 3.8-billion-parameter decoder-only language model, through a learned adapter; BioGPT supplies text embeddings for the contrastive objective. Training proceeds in two stages: the first jointly optimizes contrastive (weight 0.25) and autoregressive (weight 1.0) objectives over one epoch; the second freezes the slide encoder and updates only the adapter and language model. Training ran on 56 A100 40GB GPUs in bf16 with AdamW.

On pan-cancer detection by linear probing, PRISM2 diagnostic embeddings reach 0.976 AUC against 0.956 for its own base embeddings, 0.947 for PRISM, and 0.931 for TITAN, with only a modest drop to 0.967 on a rare-cancer subset. Biomarker prediction reaches 0.854 AUC on internal MSK data and 0.784 on TCGA. For survival, fine-tuned embeddings reach a 0.809 C-index on MSK colorectal recurrence-free survival against 0.773 for a survival specialist model. Weights are gated on HuggingFace under CC BY-NC-ND 4.0, restricted to non-commercial research, with clinical deployment and diagnostic use explicitly prohibited.

#Applications

The prompted-inference mode is the distinguishing capability: a pathology group can ask a slide direct diagnostic questions or request a draft report without assembling a labeled training set, which is the main bottleneck for narrow task-specific models. The embedding mode serves the conventional workflow — screening and triage, biomarker prediction, subtype classification, and prognostic modeling — with the diagnostic embedding preferred for diagnosis-adjacent tasks and the base embedding for broader ones. Because the model consumes pre-extracted Virchow2 tile embeddings rather than raw slides, institutions that already run tile encoders can adopt it without re-processing their archives.

#Impact

PRISM2 makes a concrete case that the text pathologists already write is the most scalable supervision signal available for whole-slide understanding, and reaching clinical-product accuracy without task-specific training marks a shift in what slide-level foundation models are expected to do out of the box. Publication in Nature Medicine and the release of gated weights give it more standing than most models in this space. Real constraints remain: the training corpus is dominated by one institution's specimens and scanner fleet, the license bars commercial and clinical use, evaluation is retrospective, and the authors are explicit that the model is not a conversational agent and must not be used for diagnosis.