MV-Hybrid

MV-Hybrid is a backbone architecture for pathology vision foundation models (VFMs) that interleaves state space model (SSM) layers with vision transformer layers, built by Deepnoid with the Department of Hospital Pathology at The Catholic University of Korea and presented as an oral at the MICCAI 2025 COMPAYL workshop.

Spatial transcriptomics measures gene expression in tissue context, but its cost keeps it out of routine practice. Predicting spatial gene expression from an H&E slide is the cheap substitute, and it is the task where pathology VFMs — UNI, Virchow, and their ViT-based peers, all pretrained on millions of slides — fall furthest short of clinical usefulness. The authors argue the bottleneck is architectural rather than a matter of data volume: molecular phenotype correlates with slow, spatially extended morphological variation, and self-attention is biased toward high-frequency detail, so a ViT tends to latch onto site-specific texture that does not transfer across studies.

Their fix follows from a frequency analysis of state space models. An SSM whose state matrix has negative real eigenvalues has a uniform frequency cutoff and a strong low-frequency bias, unlike the modality-dependent cutoffs of complex eigenvalues. MV-Hybrid puts twelve MambaVision-based SSM layers first and twelve standard ViT layers on top, so low-frequency structure is captured before attention refines it. Six backbones — MV-Hybrid, a Hydra hybrid, two pure SSM models, and two ViT baselines — were then pretrained identically and evaluated identically, which makes this one of the few architecture-controlled comparisons in the pathology VFM literature.

#Key Features

• Hybrid SSM-to-ViT stack: The first twelve layers pair a MambaVision sequence mixer with an EinFFT channel mixer; the second twelve are vanilla ViT blocks. The MambaVision backbone is made isotropic rather than hierarchical so it is compatible with DINOv2 pretraining.
• Low-frequency inductive bias: Negative real eigenvalues in the SSM give a sharp, uniform frequency cutoff, biasing features toward the coarse morphology that tracks molecular phenotype instead of scanner- and site-specific high-frequency texture.
• Architecture-controlled benchmarking: All six backbones share the same embedding dimension of 384, the same colorectal pretraining corpus, and the same DINOv2 recipe, so downstream differences are attributable to architecture alone.
• Frozen-feature evaluation: Downstream tasks use ridge regression, linear and k-NN probing, and prototype-based models on frozen patch embeddings, measuring representation quality rather than fine-tuning capacity.
• Leave-one-study-out robustness: Models are also tested by holding out an entire study source, exposing how much of a backbone's accuracy comes from batch-specific cues.
• Efficient at scale: At 30.9M parameters and 8.4 GFLOPs, the model is smaller and cheaper than the 24-layer ViT baseline (43.0M, 12.2 GFLOPs) and processes 512x512 inputs faster (231 versus 219 images/second).

#Technical Details

All backbones were pretrained with DINOv2 for 200 epochs at a learning rate of 2.5e-3 and batch size 1,536 on 756,000 tissue patches of 256x256 pixels, tiled from 40 colorectal cancer whole-slide images sampled class-stratified from the HunCRC and IMP-CRS-2024 collections. Biomarker evaluation uses paired spatial transcriptomics and H&E data from HEST: the official HEST-Benchmark colon and rectum splits with the top 50 highly variable genes, plus a curated HEST-Extended set of 54 samples from eight study sources scored on the top 200 highly variable and top 200 high-mean highly variable genes.

MV-Hybrid leads every HEST-Benchmark metric — Pearson correlation overall and for the top 10 genes, mean absolute error, and mean squared error. Under leave-one-study-out evaluation it reaches correlations of 0.138 and 0.212 on the two HEST-Extended gene sets, 42% and 71% above the best ViT, and it degrades least when moving from random splits to held-out studies. Beyond expression prediction it matches or beats the ViT baselines on MSI classification from TCGA-CRC slides, polyp and adenoma patch classification on MHIST and UniToPatho, nine-class zero-shot retrieval on NCT-CRC-100K, and prototype-based survival prediction on TCGA-CRC.

#Applications

The intended use is as a drop-in feature extractor wherever a pathology VFM already sits: predicting spatially resolved expression of clinically relevant genes from routine H&E, screening for microsatellite instability, retrieving morphologically similar patches, and building prognostic models. The robustness result is the practically important one, since a slide-based biomarker model is only useful if it survives a new scanner, laboratory, and patient population.

#Impact

MV-Hybrid is evidence that pathology VFM progress is not purely a scaling problem, and that a frequency-domain argument about architecture predicts which backbone generalizes across sites. Its limits are equally clear. The pretraining corpus is deliberately small and colorectal-only — a controlled comparison, not a competitive foundation model run — so the reported numbers say which architecture is better under matched conditions, not how the model compares with production VFMs trained on hundreds of thousands of slides. No pretrained checkpoint is released; the repository provides the pretraining and evaluation pipeline under a CC BY-NC-SA 4.0 licence, which rules out commercial use and requires derivatives to carry the same terms.