UMPIRE

Multimodal pretraining transformed computational pathology by pairing tissue images with text: PLIP with social-media captions, CONCH with more than a million curated image-caption pairs. Text is a convenient supervision signal because it is abundant, but it is a description of what a pathologist already saw. It carries no information about the molecular state of the tissue underneath, which is what determines subtype, prognosis and treatment. An encoder trained only against captions inherits that ceiling.

UMPIRE, the Unified Molecule-enhanced Pathology Image REpresentation learning framework, swaps the supervision signal. Developed at Fudan University with a collaborator at Shanghai Jiao Tong University, it aligns a pathology image encoder against measured gene expression from spatial transcriptomics rather than against text. Spatial transcriptomics supplies exactly the pairing that makes this possible: each spot on a slide has both an image crop and a transcriptome, at a spot size close to the scale of an individual tissue patch.

The obstacle is scale. Paired image-expression data is orders of magnitude scarcer than image-caption data, so UMPIRE splits training in two — a gene encoder pretrained on a large corpus of unpaired expression profiles, then contrastive alignment against a pretrained image encoder over the much smaller paired set.

#Key Features

• Gene expression as the training signal: The alignment target is measured transcription rather than a caption, which the authors characterise as a robust, task-agnostic signal because it does not depend on what a human chose to write down.
• Visiumformer gene encoder: A BERT-style transformer over a 20,310-gene vocabulary that normalises each gene by its corpus-wide mean, sorts genes into descending expression order, and treats the ranked list as a token sequence of length 1500.
• Two ways to adapt: A full fine-tuning variant and an adapter variant that adds two small linear layers to frozen encoders, training under one percent of the parameters — the adapter is the recommended route on small cohorts.
• Query-reference expression prediction: At inference the frozen vision encoder produces a query vector, expression profiles from training form a reference database, and the prediction is a cosine-similarity-weighted combination of nearest references, which sidesteps regressing thousands of genes directly.
• Improves encoders it does not replace: Alignment is applied on top of existing pathology encoders, so the gain accrues to whichever tile encoder a lab already uses.

#Technical Details

Visiumformer is a 12-layer transformer pretrained with masked language modelling at 15% masking on ViSTomics-4M — 3.94 million Visium spatial transcriptomics profiles gathered from 1,363 slides across 180 public datasets — for one million steps at a global batch size of 256. Alignment uses 697K image-expression pairs filtered from HEST, with a symmetric contrastive loss over a shared 512-dimensional space, ten epochs at batch size 512 on four A800 GPUs. The paper's vision encoders are Phikon (ViT-B/16, 86M parameters) and UNI (ViT-L/16, 307M); the repository later added CONCH-aligned weights as well.

On full-dimensional gene expression prediction across liver, prostate and HER2-positive breast cohorts, the fine-tuned model averages a Pearson correlation of 0.2902 for the top 50 highly variable and highly expressed genes, against 0.2028 for BLEEP and 0.0919 for ST-Net — a 42.9% improvement over the strongest contrastive baseline, rising to 83.8% on the HER2-positive data, which was collected on a different sequencing platform from the pretraining corpus. Linear probing lifts Visiumformer's balanced accuracy by 28.2% on dorsolateral prefrontal cortex layer classification, and the vision encoders by up to 42.3%. For whole-slide mutation-state prediction over four genes in TCGA-LUAD, alignment adds up to 13.7% AUC and 7.7% F1 to the original encoder. Replacing the contrastive loss with a reconstruction objective degrades image-side performance, indicating the contrastive formulation is doing the work.

#Applications

The output is a pair of encoders usable wherever molecular context matters: predicting spatially resolved expression from an H&E image without sequencing, classifying spots or patches by tissue type, and calling mutation status from a whole slide through a multiple-instance head. Groups that already run a pathology tile encoder can align it once and reuse the result, and the adapter variant makes that viable without large compute.

#Impact

UMPIRE establishes that molecular measurements are a usable supervision signal for pathology representation learning, not merely a downstream prediction target, and that the benefit survives transfer to a sequencing platform absent from pretraining. The work appeared as an arXiv preprint in December 2024 and was published in Pattern Recognition in 2026. Tokenization and inference code plus the aligned checkpoints for Phikon, UNI and CONCH are available, the weights distributed through a request-gated Google Drive folder; the repository carries no license file, and the full pretraining code and the ViSTomics-4M corpus have not been released. The paired data remains small relative to image-caption corpora, pretraining is confined to the Visium platform, and the underlying vision encoders carry their own access restrictions.