HistoPlexer

Imaging mass cytometry can resolve dozens of proteins in place on a tissue section, but a run covers a hand-picked region of roughly one square millimetre, consumes a section of the block, and says nothing about the rest of the slide. The H&E section is the opposite: cut for every case, covering the whole tissue, and already carrying the morphology a pathologist uses to tell a lymphocyte from a melanocyte. HistoPlexer treats that morphology as a conditioning signal. A conditional GAN takes an H&E tile and emits an 11-channel image in which each channel is one protein's spatially resolved expression profile — tumour markers (MelanA, S100, gp100, SOX10), lymphocyte markers (CD3, CD8a, CD20, CD16), the endothelial marker CD31 and two antigen-presentation markers (HLA-ABC, HLA-DR).

The hard part is that the supervision is never pixel-aligned. H&E and IMC come from consecutive sections of the same block, so a given cell is displaced, deformed or simply absent between the two planes, and a plain L1 loss penalises the generator for a shift the sample preparation introduced. HistoPlexer relaxes that constraint with a Gaussian pyramid loss comparing generated and ground-truth multiplexes across several scales rather than pixel for pixel, adds a patch-wise contrastive loss so corresponding patches sit closer in embedding space than distant ones, and trains against a least-squares GAN objective. Predicting all eleven markers with one generator lets sparse immune channels borrow morphological context from abundant ones.

It was built at ETH Zurich with the University Hospital Zurich, the University of Zurich and the University of Basel on melanoma samples from the Tumor Profiler Consortium, and first circulated as Multi-V-Stain before being renamed. It shares its goal with VirtualMultiplexer, ROSIE and MIPHEI-ViT, and takes the serial-section route: H&E and IMC from consecutive cuts of one block, paired by template matching alone.

#Key Features

• Eleven markers from one generator: Joint prediction reproduces ground-truth pairwise Spearman correlations between markers with roughly an order of magnitude lower error than separate per-marker models, making the output usable for co-localization analysis rather than eleven unrelated heatmaps.
• Losses designed for serial sections: The Gaussian pyramid and patch-wise contrastive terms score similarity at multiple scales and in embedding space, absorbing the slice-to-slice displacement that makes strict pixel supervision misleading.
• Whole-slide inference: A tile-based pipeline segments tissue by Otsu thresholding, batches 1,024 × 1,024 tiles, optionally applies Macenko stain normalization, and stitches predictions into slides of up to 100,000 × 100,000 pixels.
• Immune phenotyping from H&E alone: Intratumoral and stromal CD8+ T cell densities read off the generated multiplex separate immune-hot from immune-cold tumours at an F1 of 0.873 and a macro-average AUROC of 0.845.
• Applies where no multiplex ground truth exists: The generator trained on Tumor Profiler melanoma was run unchanged over TCGA–SKCM slides, a cohort with H&E only.

#Technical Details

The translator is a fully convolutional U-Net with a six-block strided encoder and a five-block nearest-neighbour-upsampling decoder; the discriminator is six spectrally normalised convolutional blocks. Training used 336 template-matched H&E–IMC region pairs from 78 metastatic melanoma patients, split at patient level into 231 training and 105 test regions, with 1,024 × 1,024 H&E tiles mapped to 256 × 256 IMC targets over 100 epochs. Against Pix2pix and PyramidP2P the multiplexed setting gives the best MS-SSIM and PSNR and the lowest sliding-window RMSE. Under the HYPE protocol, experts misclassified generated images 61.6% of the time for lymphocyte markers and 72.8% for tumour markers, above the 50% threshold for perceived realism. A variant conditioned on features from the UNI pathology foundation model was evaluated as an ablation, and end-to-end training without those features gave the lower MSE and better preserved biological relationships.

#Applications

The practical draw is tissue economy and coverage: a routine diagnostic slide yields a marker panel that would otherwise cost an extra section and an antibody run, across the whole tissue rather than a few chosen regions. That supports CD8+ T cell mapping and immune-subtype stratification relevant to immunotherapy decisions. On TCGA–SKCM, adding features from the generated multiplexes to H&E features improved disease-specific survival prediction by 3.18% in time-dependent C-index and immune subtype classification by 17.02% in weighted F1 — signal an H&E-only predictor misses.

#Impact

HistoPlexer shows that a jointly predicted, spatially coherent protein panel — not just individual marker heatmaps — can be inferred from routine histology and used downstream on cohorts where multiplex imaging was never performed. The authors are explicit about the limits: it was trained on one tumour type and eleven markers, morphologically similar lymphocytes get confused, so CD20 is sometimes predicted where CD3 and CD8a belong, and they describe the work as a proof of concept, with a universal model spanning cancer types still an open problem. Source code and a trained checkpoint are released under a non-commercial, no-derivatives licence and archived on Zenodo, and the Ultivue whole-slide immunofluorescence dataset generated here is public on Hugging Face; the Tumor Profiler training data remain available only by application to the consortium.