Histopathology-Molecular Alignment

Hematoxylin and eosin (H&E) whole-slide images are produced routinely and at cohort scale, but they carry no direct molecular readout. Bulk RNA-seq resolves the transcriptome but is costly and rarely available for archival tissue. This model bridges the two modalities: it trains a lightweight alignment module on top of two frozen, pretrained foundation models — a histopathology slide encoder and a bulk RNA-seq encoder — so that morphology and molecular state are mapped into a shared embedding space. Once aligned, the system supports open-vocabulary molecular prompting: an H&E slide can be queried with an arbitrary gene-set signature to predict pathway activity, without sequencing the sample or retraining the network end to end.

The approach is deliberately data-efficient. Rather than fine-tuning large backbones on paired histology-transcriptomics data, it keeps both encoders fixed and learns only a compact projection head that ties their representations together with contrastive learning. This makes alignment tractable on a multi-cancer cohort of 1,720 cases, a scale far smaller than what end-to-end multimodal training typically requires.

The model was developed at AstraZeneca by a team spanning Computational Pathology and Biomarkers in Munich and Early Oncology and Translational Medicine in Cambridge, led by Dominik Winter. It is described in an arXiv preprint that has not yet undergone peer review.

#Key Features

• Frozen dual-backbone design: Only a lightweight alignment head is trained; the histopathology slide encoder and the bulk RNA-seq encoder remain frozen, keeping the training footprint small and data-efficient.
• Open-vocabulary molecular prompting: At inference, an H&E slide can be queried with any gene-set signature to score pathway activity, so new molecular questions require no additional sequencing or retraining.
• Label-aware contrastive alignment: A symmetric InfoNCE objective is combined with a supervised contrastive term that treats samples of the same cancer subtype as positives, sharpening the joint embedding.
• Morphologically grounded predictions: Programs with clear histological correlates, such as cell-cycle and immune signatures, are recovered most faithfully, while diffuse metabolic pathways are predicted less well.

#Technical Details

The histopathology branch uses THREADS slide embeddings (1024-dimensional, built by aggregating CONCH vision-transformer patch features through attention multiple-instance learning), and the molecular branch uses BulkFormer, a bulk RNA-seq encoder with self-attention over roughly 20,000 gene tokens producing 512-dimensional embeddings. Lightweight projection heads map both into a shared space, trained with a CLIP-style symmetric InfoNCE loss plus a supervised contrastive term (weight 0.15, temperature 0.07). The cohort spans nine cancer types — non-small-cell lung (245), triple-negative breast (112), renal cell (195), colorectal (187), ovarian (216), endometrial (400), pancreatic (42), prostate (232), and bladder (92) — for 1,720 paired cases. Cross-modal retrieval reaches R@5 of 56.0% and R@10 of 71.9%, a 25-fold improvement over baseline alignment. Predicted pathway activity attains R-squared above 0.5 for cell-cycle programs and 0.50-0.75 for immune-related hallmarks, dropping to 0.20-0.27 for metabolic pathways. In a retrospective analysis of the POSEIDON trial, predicted signatures stratified overall survival in a directionally consistent manner (hazard ratio 0.70, p = 0.118).

#Applications

The model targets computational pathology and translational oncology workflows where slides are plentiful but molecular data are sparse. Because pathway activity can be read directly off routine H&E, researchers can mine archival slide collections for transcriptomic hypotheses without new sequencing, prioritize cases for confirmatory assays, and explore biomarker signatures retrospectively across large cohorts. Open-vocabulary prompting is especially useful when the gene sets of interest are defined after slides are digitized, and the retrospective trial analysis illustrates a path toward morphology-based patient stratification.

#Impact

This work shows that pairing frozen histopathology and RNA-seq foundation models with a small, contrastively trained alignment head is enough to connect tissue morphology to transcriptome-wide pathway activity, an alternative to data-hungry end-to-end multimodal training. It extends the CLIP-style joint-embedding paradigm, well established for image-text pairs, into the histology-transcriptomics setting and demonstrates open-vocabulary querying of gene programs from slides alone. The results are in-silico and, for the survival analysis, based on a single retrospective trial cohort; the finding is a preprint awaiting peer review, no public code or weights accompany it, and the model has not been validated for clinical or diagnostic use.