PathOmCLIP

Spatial transcriptomics places expression profiles onto a histology image, but running it on patient material is expensive and rarely feasible inside a clinical trial. H&E-stained slides, by contrast, are generated routinely and sit in archives attached to decades of outcome data. Predicting expression from those images would make the archives queryable — but the paired data needed to train an image-to-expression model is exactly what is scarce.

PathOmCLIP, from Yongju Lee and Aviv Regev at Genentech with collaborators at Princeton and Yale, works around that scarcity by starting from two models already pretrained at scale: Prov-GigaPath as the histology encoder and scFoundation as the expression encoder. scFoundation is chosen because it does not bin the expression profile, suiting Visium data where each spot aggregates 10 to 100 cells. A contrastive objective aligns the two embedding spaces, so the limited paired data is spent learning a correspondence rather than representations.

The paper's second observation is that per-patch alignment is not enough: tumor biology is a neighborhood property, and a spot's expression depends on the multicellular architecture around it. PathOmCLIP therefore inserts a LocalTransformer — a set transformer over the k nearest image patches — both inside the contrastive stage and again in the prediction head, and the ablations show it is what makes the results robust.

#Key Features

• Two pretrained foundation models, one joint space: A symmetric InfoNCE loss aligns GigaPath patch embeddings with scFoundation spot embeddings, keeping scFoundation frozen while the pathology encoder and the LocalTransformer train.
• Neighborhood context by set transformer: The LocalTransformer takes the query patch plus its k-nearest neighbors as an unordered set with no positional encoding and returns a contextualized embedding for the query alone, so tissue architecture informs the prediction without imposing a grid.
• Two-stage training: Stage one produces a pretrained LocalCLIP; stage two freezes it and trains a second LocalTransformer with a one-layer MLP under MSE loss against measured expression.
• Spatial structure preserved, not just correlation: Leiden clusterings of predicted and measured profiles are compared by adjusted mutual information, catching the failure mode where a model scores well on per-gene correlation while flattening spatial pattern.
• Evaluation on patient-stratified splits: Training and testing follow HEST-1K's own patient-stratified splits, so no patient contributes to both sides.

#Technical Details

Data is the July release of the HEST-1K 10x Visium benchmark: 224×224 H&E patches covering 112×112 µm, matched to spot expression, across clear cell renal cell carcinoma, prostate, rectal, pancreatic and colonic adenocarcinoma. Raw counts are log-transformed and gene symbols unified per tumor type; highly variable gene panels of 50, 200, 500, 1,000 and 2,000 genes are used, the 50-gene panel from HEST and the rest computed per tumor type with the slide as batch key. Training ran on four A100 80GB GPUs under Fabric with bfloat16, using four local neighbors at batch size 64 for the contrastive stage and 50 at batch size 4,096 for prediction.

On ten held-out ccRCC slides, average Pearson correlation with measured expression is 0.288 ± 0.036 at 50 genes, falling to 0.128 ± 0.025 at 2,000 genes. The comparators at 50 genes are 0.219 for a directly fine-tuned GigaPath, 0.190 for mclSTExp and 0.116 for BLEEP, and PathOmCLIP leads at every panel size. The fine-tuned GigaPath baseline is the informative one: it beats both dedicated histology-to-expression methods on correlation, yet its Leiden clusterings diverge visibly from ground truth while PathOmCLIP's track it — a difference the paper attributes to the LocalTransformer acting on image embeddings rather than expression profiles. Performance varies by tumor type: 0.464 ± 0.016 for PAAD and 0.381 ± 0.014 for PRAD at 50 genes, against 0.199 for READ.

#Applications

The intended setting is retrospective: applying the trained model to archived H&E slides from cohorts with long-term outcome data, where spatial transcriptomics was never run and cannot be. Genes where PathOmCLIP separates most from the fine-tuned pathology baseline include proliferation markers (LY6E, STK39, NOTCH3) and angiogenesis genes (EGFL7, HTRA1), and its predicted maps preserve spatial variation the baseline flattens into near-uniform expression. That makes the output usable for biomarker hypothesis generation and for stratifying regions within a slide.

#Impact

PathOmCLIP states a design pattern for data-poor multimodal problems: align two independently pretrained foundation models rather than training a joint model on scarce paired data, and add locality where the biology is local. Its scope should be read carefully. The deployed artifact is a family of per-tumor-type models with tumor-type-specific gene panels, not one cross-tissue model, and absolute correlations remain modest — under 0.3 even on the best ccRCC panel — so predictions are best treated as a spatial hypothesis rather than a measurement. All evaluation stays within HEST-1K's own splits, with no external cohort. The work remains a preprint, and no code or trained weights have been released.