ALTER

Computational pathology has largely been built around a single modality. Slide-level foundation models such as UNI and CONCH learn from histology alone, and the multimodal work that followed usually pairs whole-slide images with exactly one partner — either bulk genomics or free-text reports. That leaves two practical problems. Gigapixel slides make naive fusion computationally punishing, and real clinical cohorts are ragged: a case may have a slide and a report but no sequencing, or omics without a narrative write-up.

ALTER, from Peking University and the Hong Kong University of Science and Technology, is a pretraining framework designed around that raggedness. It jointly models three modalities — whole-slide images, bulk gene-expression profiles, and diagnostic reports — in a single sequence transformer whose attention is computed only over whichever modalities are present. The "any-to-any" framing is literal: the same model can be pretrained on partial modality combinations and then deployed on unimodal, bimodal, or fully tri-modal downstream tasks without imputing the missing inputs.

The result is a computational-pathology model evaluated across four task families rather than one, spanning prognosis, classification, molecular inference, and text generation. The paper is a preprint and has not yet been peer reviewed.

#Key Features

• Three modalities in one encoder: Slides, transcriptomes, and reports share a universal sequence transformer, so cross-modal structure learned during pretraining remains available even when only one modality is supplied at inference.
• Missing-modality robustness by construction: Attention is computed over the available subset only, so incomplete cases need no imputation or placeholder substitution — a common source of noise in concatenation-style fusion.
• Modality-specific mixture of experts: After a shared self-attention stage, per-modality expert modules decouple the fused latent space, accommodating the spatial statistics of slides, the sparse tabular structure of omics, and the semantics of clinical text.
• Three-tiered pretraining objective: Masked modeling within each modality, CLIP-style contrastive alignment across all three modality pairs, and a cancer-type triplet loss at the sample level.
• Task-agnostic deployment: Downstream heads are unconstrained — cross-attention over the CLS tokens for multimodal tasks, simple MLPs otherwise — with fusion layers frozen for unimodal tasks and fine-tuned for multimodal ones.

#Technical Details

Pretraining uses 6,850 matched WSI-omic-report triplets drawn from 29 TCGA cancer types; four further cancer types are excluded entirely from pretraining and reserved for downstream validation, preventing leakage. Slide patches are embedded with a frozen UNI encoder and grouped into regions before entering the transformer, gene expression is organized by biological pathway rather than as isolated genes, and reports are encoded with BioBERT over a 512-token context. The combined loss weights masked-language-modeling, contrastive, and triplet terms.

Evaluation spans four task categories across ten public datasets. On survival prediction over four TCGA cohorts (UCEC n=480, STAD n=317, KIRP n=284, KIRC n=218), ALTER improves overall C-index by 4.1% over the next-best method. On cancer subtyping across BRACS, PANDA, TUPAC-16, and UBC-OCEAN it leads on three of four benchmarks, improving overall AUC by 2.7% and F1 by 5.3%. Predicting mutation status from lung adenocarcinoma slides alone (n=412) yields AUCs of 0.809 for TP53 and 0.811 for EGFR. On report generation from PatchGastricADC22 (991 slides, 262,777 patches) it reaches BLEU-1 through BLEU-4 scores of 0.628, 0.547, 0.491, and 0.450, ahead of WSICaption and HistGen. All baselines use the same UNI patch features, isolating the contribution of the pretraining framework.

#Applications

ALTER fits translational pathology groups working with TCGA-style archives where slides, sequencing, and reports coexist but are unevenly available across patients. A single pretrained backbone supports prognostic modeling for treatment stratification, automated subtype classification, inference of driver-gene mutation status from morphology alone, and draft report generation — all from whatever data a given case happens to carry. The missing-modality design is the practically important part: it removes the usual requirement that a multimodal model be applied only to fully paired cohorts.

#Impact

The paper's clearest finding is that multimodal pretraining benefits unimodal downstream tasks: cancer subtyping from slides alone improves after tri-modal pretraining, and freezing the fusion blocks preserves that transferred knowledge. That argues for multimodal pretraining even when deployment will be image-only, which is the common clinical case. Limitations are real. Pretraining draws on TCGA alone, so multi-center generalization is untested and the authors identify it as necessary future work; validation is retrospective and computational throughout; and no code or model weights have been released, which leaves the framework reproducible only from the paper's description.