Loading…
Histopathology foundation model aligned to spatial transcriptomics by a cross-modal ranking loss, embedding H&E patches without gene input.
No providers recorded yet. Browse all providers
A spatial transcriptomics slide pairs every tissue spot with an image patch and a gene expression profile, inviting the obvious approach: push matched image and gene features together in a shared latent space. But expression vectors are extremely sparse — more than 15,000 genes per sample, around 80% of values zero — and spot counts carry spatially dependent noise, so distances between gene features are unreliable even when their relative ordering is not. RankByGene is built on that distinction: rather than matching cross-modal distances, it requires that for any anchor spot, the ranking of its similarities to other spots agree between the image space and the gene space.
Developed at Stony Brook University with collaborators at Harvard Medical School, Stanford University, and Columbia University, it adapts the UNI ViT-L/16 pathology backbone rather than pretraining from scratch, reshaping its features so that morphology encodes cell-functionality signal image-only self-supervision leaves on the table.
Gene expression is therefore needed only during training. At inference the teacher image encoder alone embeds any H&E patch to a 1024-dimensional vector, so the representation applies to image-only cohorts — which is most of digital pathology.
The image encoder is a UNI ViT-L/16 with a three-layer MLP projection head; a separate three-layer MLP encodes the per-spot gene vector, and both map into a shared 1024-dimensional space. Training minimizes a weighted sum of an InfoNCE gene–image contrastive loss, the ranking consistency loss (weight 5), and the intra-modal distillation loss (weight 1), at temperature 0.1 and EMA momentum 0.96, for 100 epochs at batch size 64 on a single 48 GB GPU. Enumerating every triplet in a batch costs O(N³); sampling O(N) pairs per anchor brings that to O(N²) with no meaningful accuracy loss.
Alignment uses breast (36 samples) and lung (6 samples) spatial transcriptomics from HEST-1k, restricted to prognosis-related Human Protein Atlas gene panels — 447 genes for breast, 1,916 for lung — after L1 normalization, log transformation, and 8-neighborhood smoothing. Evaluation runs on held-out Visium slides (Breast-ST1, 4,898 spots; Breast-ST2, 3,813 spots; four lung slides totaling 1,831 spots) plus TCGA and BCNB cohorts disjoint from the alignment data. Pearson correlation for expression prediction improves 9–34% over the best baseline on top-250 highly expressed genes and 7–35% on cancer-specific panels. With frozen features aggregated by attention-based MIL, slide-level AUC reaches 0.829 for BCNB progesterone-receptor status and 0.861 for TCGA-LUAD STK11 mutation status, and survival concordance indices are 0.681 on TCGA-BRCA and 0.595 on TCGA-LUAD.
The immediate audience is computational pathology groups holding far more H&E slides than paired spatial transcriptomics. Because alignment happens once and inference needs only the image, the released breast and lung checkpoints embed archival slides from any cohort — as the paper's own work on TCGA-BRCA, TCGA-LUAD, and BCNB does — feeding MIL classifiers for receptor status and driver-mutation prediction, survival models, and spot-level expression prediction. For groups already running an ST prediction framework, encoder substitution is the cheapest path to adoption: swap the feature extractor, keep everything downstream.
RankByGene belongs to a growing line of work that treats molecular measurements as a supervisory signal for histology encoders rather than as a prediction target, alongside BLEEP, HEST-FT, and NH²ST. Its contribution is the argument that ranking, not distance, is the right invariant to carry across two modalities that agree on ordering but not on scale — supported by ablations showing the ranking and distillation terms to be complementary, and the gain to survive a change of backbone. The work was accepted by IEEE Transactions on Medical Imaging in 2026. Its scope is narrow by design: only breast and lung checkpoints exist, aligned on small HEST-1k subsets, so transfer to other tissues is unestablished. The code is MIT-licensed on GitHub, while the checkpoints ship from a Google Drive folder with no stated license or versioned release, and training from scratch requires the gated UNI backbone weights.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.