VirTues

Spatial proteomics — imaging mass cytometry (IMC), CODEX, Orion, MIBI — measures dozens to hundreds of proteins in intact tissue at subcellular resolution, and has become central to studying the tumour microenvironment. Its flexibility is also what fragments the field: every study picks its own antibody panel, protocol and platform, so a model fit to one cohort cannot read another. Conventional image encoders assume a fixed channel stack and have no way to relate a marker they were never trained on.

VirTues (Virtual Tissues) attacks that problem by giving each channel a biological identity rather than an index. Every marker is embedded through its amino acid sequence using a protein language model, which places heterogeneous panels in a shared space and lets a single pretrained backbone ingest arbitrary marker combinations — including proteins absent from pretraining. From that one backbone come patch, cell, niche and tissue summary tokens supporting marker reconstruction, cell segmentation and typing, niche annotation, case retrieval, patient stratification and spatial biomarker discovery, with no task-specific fine-tuning.

The model was developed in Charlotte Bunne's lab at EPFL with collaborators at ETH Zurich and the University of Zurich, released as a preprint in January 2025 and published in Nature in 2026. It sits alongside KRONOS as one of the marker-aware spatial proteomics foundation models, and is the baseline-beating entry in that comparison.

#Key Features

• Sequence-grounded marker tokens: Each marker's ESM-2 embedding is fused with its spatially patched image content, so panels that share no antibodies still land in one representation space and new markers inherit meaning from sequence similarity.
• Factorized marker and spatial attention: Attention is split into a marker axis and a spatial axis instead of running jointly over both, avoiding the quadratic blow-up that makes standard vision transformers impractical on high-plex imaging.
• Hierarchical summary tokens: Learnable patch summary tokens are convolved with segmentation masks into cell tokens and aggregated into niche and tissue tokens, exposing structure from subcellular scale up to whole biopsies.
• Zero-shot transfer across panels and platforms: Cohorts, marker panels and imaging technologies withheld entirely from training are handled without retraining, including technologies never seen during pretraining.
• Virtual staining: Fully masked channels can be inpainted, letting a study impute markers it never measured while preserving downstream cell type calls.

#Technical Details

VirTues is a purpose-built vision transformer trained as a masked autoencoder. Images are processed crop-wise into three-dimensional grids of tokens indexed by marker and spatial position; marker tokens come from the 150M-parameter ESM-2 checkpoint applied to each protein's canonical UniProt sequence and are fused into the image tokens by linear projection and addition. During pretraining, tokens are masked independently per channel at a channel-wise ratio of 60–100%, and the decoder reconstructs each channel from the unmasked tokens of that channel plus all patch summary tokens. The core corpus is 15 IMC cohorts spanning 8 organ sites, 3,102 patients and 146 markers, extended to 32 cohorts across four imaging technologies, more than 5,100 patients and 239 markers.

Reconstruction reaches Pearson r of 0.800 under independent masking, 0.700 when an entire marker channel is hidden and 0.668 when whole niches are occluded. Linear probes on cell tokens improve macro-F1 for cell typing by 5.81% over KRONOS and 65.79% over a channel-agnostic masked autoencoder, with zero-shot macro-F1 of 0.691 on CODEX and 0.607 on Orion data. Segmentation heads trained on held-out cohorts beat Cellpose, InstanSeg and StarDist at IoU 0.5 on 8 of 9 datasets. Code is MIT-licensed; the virtues-sp32 weights are CC BY-NC 4.0 and a 31-dataset virtues-sp31 checkpoint is MIT.

#Applications

The clearest payoff is biomarker discovery. In the NeoTRIP triple-negative breast cancer trial, unsupervised clustering of pre-treatment cell embeddings yielded four spatial signatures whose combination predicted anti-PD-L1 chemo-immunotherapy response at AUROC 0.823, exceeding the published spatial predictor for that cohort and immune-ratio baselines. Transferred to an independent cohort, the same signatures stratified disease-free survival (log-rank P = 0.0037, concordance index 0.628). Pathology and translational oncology groups also gain a single pipeline for segmentation, phenotyping, niche annotation and retrieval of clinically similar cases across archives acquired on different platforms.

#Impact

VirTues demonstrates that panel heterogeneity, long the obstacle to pooling spatial proteomics cohorts, is tractable when marker identity is encoded from protein sequence rather than channel order. Alongside the model, the authors released the harmonized spora corpus and benchmark suite, giving the field shared data and evaluation infrastructure. Real limits remain: zero-shot reconstruction degrades for markers with weak sequence relatedness to the training panels, rare cell states and unusual tissue architectures are still hard, and development and evaluation centred on IMC. The flagship checkpoint's non-commercial licence also restricts industrial use, though the MIT-licensed 31-dataset variant exists for that case.