Predicts 16-channel multiplex immunofluorescence from H&E histology using a ViT foundation-model encoder, validated on five external datasets.
No providers recorded yet. Browse all providers
MIPHEI-ViT predicts multiplex immunofluorescence (mIF) signal directly from routine H&E histology. H&E is acquired for essentially every cancer patient and shows morphology and tissue architecture; mIF resolves cell identity precisely through protein markers, but its cost and logistical demands have kept it out of routine clinical use. MIPHEI is an attempt to recover part of that molecular readout from the slide that already exists, developed by Digital R&D at Sanofi with the Centre for Computational Biology and the Centre for Statistics and Images at Mines Paris – PSL and Institut Curie.
The architecture is a U-Net-style image-to-image model whose encoder is a pathology vision transformer foundation model — H-optimus-0, a ViT-G/14 trained on hundreds of millions of tiles from over 500,000 whole-slide images. Rather than learning morphology from the training corpus alone, MIPHEI inherits it, and adapts the encoder with LoRA so that only a small fraction of weights are updated. It targets a 16-channel panel spanning nuclear content, immune lineages, epithelium, stroma, vasculature, and proliferation.
MIPHEI's distinguishing evidence is external. Trained once on the public OrionCRC colorectal cohort, it is evaluated without retraining on five independent datasets (HEMIT, PathoCell, IMMUcan, Lizard, PanNuke) that differ in staining protocol, marker panel, tissue, and magnification. That places it alongside other virtual-staining models in the catalog — UNIStainNet for chromogenic IHC, HistDiST for diffusion-based H&E-to-IHC transfer, and Cytoland for label-free light microscopy — but with a multiplexed fluorescence target and cross-cohort validation.
Training used OrionCRC, 41 colorectal-cancer whole-slide images with restained, pixel-aligned H&E and mIF, tiled at 256×256 pixels and 0.5 microns per pixel. The loss is a per-marker weighted MSE scaled by each marker's signal standard deviation, with Tanh output activations; training ran on a single A100 at batch size 16 with spatial, stain, brightness, blur, and noise augmentation, plus precomputed style-translated tiles to simulate scanner and protocol shift. An ablation across ConvNeXt, UNETR, and ViTMatte decoders and across three pathology foundation-model encoders selected the ViTMatte design with an H-optimus-0 encoder and LoRA, trained without a GAN discriminator. On the OrionCRC test set, cell-type classification from H&E alone reaches F1 scores of 0.93 for Pan-CK, 0.83 for α-SMA, 0.68 for CD3e, 0.36 for CD20, and 0.28 for CD68 — ahead of Pix2Pix, HEMIT, ROSIE, DiffusionFT, and a nuclear morphometry baseline on most markers and datasets. Performance drops on PathoCell, where a 58-marker labeling panel does not align with the predicted set, and on PanNuke, where cross-organ tissue and higher magnification impose a larger domain shift.
The immediate use is retrospective: applying cell-type-aware analysis to the large archives of H&E slides that already exist, in cohorts where mIF was never run. That enables studies relating spatial immune organization — T-cell infiltration patterns, myeloid distribution, stromal architecture — to survival or treatment response at cohort scale rather than in the small series mIF economics normally permit. It also serves as a triage step, identifying slides or regions where true mIF is worth the expense. The published caution applies: the model is a hypothesis-generation and exploratory tool, not a substitute for measured protein expression.
MIPHEI-ViT is peer reviewed and published in Computers in Biology and Medicine (2026), with code, pretrained weights via GitHub releases, a hosted demo, and preprocessed OrionCRC and HEMIT data on Zenodo. It is one of the clearer demonstrations that a general pathology foundation model transfers usefully to dense prediction rather than only to slide- and patch-level classification, and its five-dataset external protocol sets a higher generalization bar than the single-cohort evaluations common in virtual staining. The honest per-marker breakdown is equally consequential: markers whose expression is not written into nuclear morphology, such as CD20 and CD68, remain poorly recovered, which bounds how far H&E-derived immunophenotyping can go. The code and weights are released under a Sanofi non-commercial license, which restricts use to academic and non-profit work.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.