Early-fusion pathology foundation model reading H&E tiles and spatial transcriptomics as one token sequence, with or without transcripts at inference.
No providers recorded yet. Browse all providers
Most methods that pair whole-slide images with spatial transcriptomics use the molecular data only as a teacher. A contrastive objective pulls image embeddings toward matching expression profiles, or a regression head is fine-tuned to predict genes from pixels, and in both cases the trained model looks only at the image. That suits routine clinics, where H&E slides are everywhere and sequencing is rare, but it wastes the transcripts on the occasions when a lab does have them, and it reduces any fusion to concatenating features late in the network.
HistoTx, from kaiko.ai with the Biomedical Data Science Center at Lausanne University Hospital, the SIB Swiss Institute of Bioinformatics, the University of Lausanne, EPFL and thoracic-oncology clinicians at the ETOP IBCSG Partners Foundation, Vall d'Hebron University Hospital and The Royal Marsden, treats transcripts as a genuine input. Gene-count tokens are merged with image patch tokens and processed by the same transformer layers, so the two modalities interact from the first block onward. The model is trained by continuing self-supervised pretraining from a vision-only pathology foundation model, and the resulting checkpoint runs on images alone or on images plus transcripts.
The work extends an earlier supervised early-fusion transformer from the same groups to self-supervised pretraining and to mixed spatial platforms. It was posted to arXiv in November 2026. An unrelated GitHub package also called HistoTx, a ResNet-50 expression predictor, is not this model.
The morphology branch is a ViT-B/14 initialized from an internal kaiko checkpoint trained with the Midnight recipe at that size, not the publicly released Midnight weights, and it is fine-tuned during training. The expression branch is the frozen 6-layer, 10M-parameter Geneformer variant, gf-6L-10M-i2048, followed by batch normalization and a linear projection. Training uses the DINOv2 objective with the iBOT masked-image loss and a KDE regularizer, with AdamW, a cosine schedule, an effective batch size of 1,024 and up to 200 epochs. Six equally weighted data streams supply training samples. The paired streams are HEST-1k v1.1.0 Xenium and Visium data (training split only) and 122 mesothelioma samples from the ETOP BEAT-meso cohort, which have Xenium immuno-oncology (380 genes) and custom (100 genes) panels. The image-only streams are those same sources plus 7,924 TCGA diagnostic slides, included to preserve general morphology features.
The headline variant uses one token per Visium spot and 16 per Xenium tile. On HEST-Bench, image-only Pearson correlation rises from 0.31 for the starting backbone to 0.38, above SEAL applied to CONCH at 0.35. On BEAT-meso, predicting 100 held-out genes reaches 0.58 from the image alone and 0.66 when the other 380 genes are provided. A supervised early-fusion model trained only for that task reaches 0.82. Segmentation and Camelyon16 metastasis detection improve, while breast-subtyping benchmarks (BACH, BRACS, BreakHis) drop by 4 to 9 points of accuracy.
HistoTx targets translational pathology groups with archival H&E alongside some spatial profiling, such as cancer cohorts with a Xenium panel on part of the tissue. The image-only mode serves as a drop-in tile encoder with stronger expression prediction than its vision-only parent. The multimodal mode can infer genes outside a measured panel or produce joint morpho-molecular embeddings for downstream probes.
HistoTx shows that early fusion of image and transcript tokens can be learned without labels inside a standard DINOv2 pipeline, rather than requiring task-specific supervision, and that one model can absorb both imaging-based and sequencing-based spatial platforms. Its results also map out a tradeoff: molecular pretraining helps expression-linked and segmentation tasks but erodes fine morphological subtype discrimination. That gives later work a concrete target. It is a preprint, evaluation is in silico, and no code or weights have been released.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.