ROSIE

A pathologist reading a hematoxylin and eosin (H&E) slide can tell that a cell is a lymphocyte, but not whether it is a B cell or a T cell — that distinction needs an antibody stain. Multiplexed immunofluorescence (mIF) platforms such as CODEX/PhenoCycler resolve dozens of such markers in place, but they are expensive, slow, and largely absent from clinical labs, so the molecular layer is simply missing from the H&E slides already archived.

ROSIE ("RObust in Silico Immunofluorescence from H&E") treats that gap as a regression problem rather than an image-generation problem. Given a 128 × 128 pixel H&E patch, a convolutional network predicts the mean expression of 50 protein biomarkers over the central 8 × 8 pixel block; sliding that window across a sample and stitching the predictions yields a whole 50-channel image aligned to the original slide. All 50 markers are read off a shared penultimate embedding, so related biomarkers borrow each other's representations and the full panel trains better than the smaller ones the authors tried. Avoiding the adversarial objectives of earlier virtual-staining work, ROSIE optimizes a single masked mean-squared-error loss, which is what lets it scale stably to 50 markers.

The model was built by Enable Medicine with Stanford University and clinical collaborators in Tübingen, Seattle, New Orleans, and Chicago, and published in Nature Communications. It sits between purely morphological pathology foundation models and the group's later matched-modality encoder, Eva, alongside other H&E-to-mIF translators such as MIPHEI-ViT.

#Key Features

• 50-marker panel in one forward pass: A single checkpoint emits a 50-channel TIFF spanning immune, epithelial, stromal, and vascular markers — DAPI, CD45, CD3e, CD8, CD20, PanCK, aSMA, CD31, Ki67 and others — rather than one or a handful of stains.
• Masked loss over heterogeneous panels: Each training study stained a different antibody subset, so the loss covers only the markers present in a sample, letting 148 distinct biomarkers across studies train one shared 50-marker head.
• Resolution chosen at inference: The standard 8 px stride yields 3.02 µm per pixel; a 1 px stride produces 64× more overlapping predictions at 0.3775 µm per pixel.
• Phenotyping, not just pictures: Predicted expressions drive k-nearest-neighbor cell phenotyping (F1 0.61 epithelial, 0.56 T cells, 0.55 fibroblasts), graph-based tissue structure discovery (ARI 0.475 versus 0.105 for a morphology baseline), and tumor-infiltrating lymphocyte counts correlated at Pearson R 0.805 with ground truth.
• Built-in quality gating: Per-channel dynamic range and the Wasserstein distance between a test slide's intensity histogram and the training distribution flag inputs likely to generate poorly, raising average Pearson R from 0.285 to 0.312 and 0.336 when used as filters.

#Technical Details

ROSIE is a ConvNeXt-Small CNN of roughly 50M parameters, initialized from ImageNet pretraining and fine-tuned as a 50-output multitask regressor on 4 V100 GPUs. Training used 18 studies, 1,342 co-stained H&E/CODEX samples, 13 disease types and over 16 million cells. Evaluation spans four studies (485 samples, about 4.9 million cells), two of which — Stanford-PGC and UChicago-DLBCL — are also represented in training under a coverslip-level split. Across all 50 biomarkers ROSIE reaches Pearson R 0.285, Spearman R 0.352, and a sample-level C-index of 0.706, against 0.007 for an H&E-intensity baseline and 0.072 for a cell-morphology neural network. On Ochsner-CRC and Tuebingen-GEJ — clinical sites and disease types absent from training — Pearson R is 0.241 and the C-index 0.633, versus 0.319 and 0.694 on the in-distribution Stanford-PGC set. This 50M-parameter CNN outperformed substantially larger vision transformers, including a 300M-parameter ViT-L/16 pathology model, as well as a pix2pix GAN trained on the same data.

#Applications

The practical target is the archival H&E cohort that will never be assayed by mIF. ROSIE lets researchers phenotype B cells, T cells, macrophages and fibroblasts, quantify tumor-infiltrating lymphocytes, and recover stromal and epithelial microenvironments from slides already in the freezer, and it can triage which cases warrant a real antibody panel. Because the network is small, evaluating a tissue microarray core takes about five minutes on a single on-premise or consumer GPU, or on CPU.

#Impact

ROSIE demonstrates that H&E morphology encodes far more molecular signal than the few cell classes conventional image analysis extracts, at a panel size an order of magnitude beyond earlier paired-data virtual staining. Its own framing is candid: absolute correlations are modest and uneven across markers, so the authors rest downstream use on the rank metrics, and they state plainly that ROSIE is not a replacement for a full-panel CODEX experiment. All training data was acquired in-house on one H&E scanner and one PhenoCycler instrument, and a cross-platform test on Orion-imaged colorectal cancer showed some markers holding up while others degraded. Inference code is public under CC BY-NC 4.0 and weights ship through a gated Hugging Face repository, but the raw training images are withheld for licensing and privacy reasons.